[production] Remove citrix, kaspersky, rapid7 and tenable packages

[adriansr]

We've decided to pull out these experimental packages due to issues with them:

• citrix and kaspersky parsers expect data from a ODBC connection.
• rapid7 and tenable require additional parsing.

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #367 opened]

• Start Time: 2020-08-20T10:01:33.931+0000

• Duration: 10 min 43 sec

Test stats 🧪

Test	Results
Failed	0
Passed	35
Skipped	0
Total	35

[ruflin]

@adriansr Pulling these out is a bit late as 7.9 is shipped and it might be that some users already installed it. So far Kibana assumes packages which were available do not disappear. We plan to change this in 7.10.

So I'm worried by getting this PR in and shipping it, that we break some deployements. It would be better to fix the integration instead. And if want to go with the above forward, we should test on what happens if you have the package installed and it is not there anymore aftewards.

[adriansr]

@ruflin I know it's late, sorry about that.

I've tested what happens when they are removed and an agent is already using them:

• They don't show up in Installed integrations, but an Error: Internal Server Error is logged in Kibana logs for each of them when accessing the installed integrations page.
• They show up in the Configurations page:
  • Edit on the integration gives an error page: Error loading data
  • Remove the integration removes it from the agent.
• The non-removed integrations keep working, no errors in elastic-agent or Filebeat logs.

Unfortunately fixing the integration is not possible. It's an autogenerated integration from an external source and now we found out that the source is not valid. It implies creating a new integration from scratch.

Do we have monitoring to know how many agents have these integrations installed?

[ruflin]

@adriansr Thanks for testing this. Could you try even though the package does not show up in the UI, if it can still be removed through an API call. @neptunian can help you with details here.

My thinking: For the users that tried it out and do not want to see the errors forever, we can provide at least with a command to clean it up.

We do not know how many agents use it.

[adriansr]

@ruflin, it's possible to go to Integrations tab, you get a list of all used integrations including the removed ones, and remove these integrations in there. Is that equivalent to the API call?

[ph]

@ruflin, it's possible to go to Integrations tab, you get a list of all used integrations including the removed ones, and remove these integrations in there. Is that equivalent to the API call?

That should be equivalent, @neptunian could confirm.

[ruflin]

@adriansr Perhaps an alternative idea here, is to add a new package which is almost empty. Like this, on update the existing assets get removed and the user can even uninstall the package in the UI.

[adriansr]

@ruflin @ph are we ok with removing them then?

[ruflin]

@adriansr By removing you mean my suggestion before your comment or getting rid of them completely?

[adriansr]

@ruflin, I missed it.

Perhaps an alternative idea here, is to add a new package which is almost empty.

How does this work? I understand the package will still show up in the UI? Isn't that misleading for users installing this new package?

[ruflin]

@adriansr Yes, it will still show up but hopefully all the assets are removed. It allows us to give users some communication on why it was removed and probably more important for us, it makes sure things don't break. Also I hope it allows users to remove it.

[ruflin]

@skh I think you can comment on some experience with disappearing packages?

[skh]

I think it depends which assets are the problematic ones, as not all assets that were removed in the package will be removed from the system when the package is updated. (Cc @neptunian for details.)

[neptunian]

All the Kibana assets are removed, but the index templates will not be removed on an update. You will need to remove the package to do that.

@ruflin, it's possible to go to Integrations tab, you get a list of all used integrations including the removed ones, and remove these integrations in there. Is that equivalent to the API call?

I'm not sure I understand this. You should not see removed integrations in the list of installed integrations. Removing an integration is equivalent to calling the endpoint DELETE /api/ingest_manager/epm/packages/(pkg). Keep in mind you cannot remove an integration that is in use by a package policy that is in use by an agent.

[adriansr]

@neptunian , I wasn't clear.

I mean for an integration that is removed from this repo while in use by an agent:

• It is not shown in the available integrations
• It is still shown in the Integrations tab (because it's in use by an agent)
• It is still installed in the agent.
• This integration can be removed in the UI and that causes the agent to stop using it.

Is it clearer now?

[neptunian]

@adriansr Got it, thanks. If we want to go the update route, you would need to make sure to update the dataset fields to just have some basic fields that would create some simple mappings and index templates that could overwrite the current ones. Everything else should be able to be deleted in the package and it will be deleted in elasticsearch and kibana.

[ruflin]

@adriansr We seem to be stuck here a bit. By now we have branched out production-7.9 branch for the 7.9 deployment. My suggestion to move forward:

• Remove these in the production branch (this will only effect 7.10)
• We need to make sure we can deal with removed packages in 7.10 (@ph )
• We keep 7.9 as is to not break users. Not too nice for users wanting to try out the package but probably safest option.

[adriansr]

Thanks @ruflin, makes sense.

Remove these in the production branch (this will only effect 7.10)

That means I can merge this PR? Can you approve?

[ruflin]

@ph Lets make sure in 7.10 things don't break if a package is missing. @adriansr Could you file a issue in Kibana for this?

[adriansr]

@ruflin @ph I created elastic/kibana#76489, feel free to edit and ping the appropriate people if necessary.

[ruflin]

Thanks @adriansr I will rollout the new version of the registry now. Would be nice if you could check that it actually disappeared in the most recent version.