TAXII, or Trusted Automated Exchange of Intelligence Information, is a protocol designed specifically to share cyber threat intelligence.
TAXII enables organisations to share CTI by defining a single API that all upstream and downstream technology can be built to support, removing the issues of trying to support many individual API designs.
Before I kick off, it’s important to make the distinction between STIX and TAXII. Many often confuse the two, myself included only a few month ago.
STIX is a representation of threat intelligence – the content.
TAXII is a standard way to share that content – the protocol.
So how are they related? A TAXII Server must be able to handle STIX content (print it in responses, and receive it from producer).
A TAXII server can also handle other intelligence formats, in addition to STIX. However, in the following tutorial posts, I will only focus on TAXII servers and clients that use STIX 2.1 structured data. Why? 99% of the industry use TAXII to distribute only STIX data (total guess, however, I’m yet to see a TAXII server distributing anything but STIX… prove me wrong!).
Just one final note for the avoidance of doubt… one of the reasons that leads to confusion is this versioning of the standards. STIX is currently on version 2.1. TAXII, also currently on 2.1. However the versioning is completely independent and there is no coupling of the two based on version.
Now that’s clear (hopefully), at its core; TAXII has two main concepts…
- TAXII Servers: store created intelligence from and disseminate it to consumers (the TAXII Clients) via the API
- TAXII Clients: that publish intelligence and/or consume intelligence from TAXII Servers
Note, a TAXII Server and Client can be the same machine.
For example, a Threat Intelligence platform acts as a TAXII Client and consumes intelligence feeds from remote TAXII Servers. The TIP also acts a TAXII server for downstream security tools connecting to the TIP to poll for curated intel.
TAXII Services
The TAXII 2.1 specification defines two primary services to support a variety of common intelligence sharing models:
- Collections: An interface to a server-provided repository of objects that allows a producer to serve consumers in a request-response template.
- Channels: Allows the exchange of information according to a publish-subscribe model.
For the more technically inclined, a good equivalent is to think of Collections as a REST API and Channels as webhooks.
In reality, the Channels service is not yet defined and I am not really sure why OASIS decided to include it in the published specification. As such, Channels will be touched on minimally in this tutorial.
Collections and Channels can be organized in different ways.
The search for information on a TAXII server depends on what you are looking for and how you want to receive it. Generally the design of Collections and Channels on a TAXII Server will look something like this
TAXII Clients contain the logic to consume data from and publish data to Collections (request/response) or Channels (streamed) via the TAXII Servers API.
A TAXII Client might just be a script making API calls to the TAXII Server to retrieve data, though a few fully fledged TAXII Clients products with more advanced logic to interact with TAXII APIs exist, some of which I will show you in this tutorial.
Before I get into those, I will demonstrate these concepts by going through the TAXII APIs.
To do this, lets jump right in and install a TAXII Server.
Medallion TAXII server
Medallion is an an open-source TAXII implementation build by OASIS.
Medallion has been designed to be a simple front-end REST server providing access to the endpoints defined in the TAXII 2.1 specification.
It is important to note that medallion was designed as a prototype and a reference implementation of TAXII 2.1 – it was not intended for production use.
Medallion can be installed very simply;
git clone https://github.com/oasis-open/cti-taxii-server
cd cti-taxii-server
git checkout v2.0.1
python3 -m venv cti-taxii-server
source cti-taxii-server/bin/activate
pip3 install medallion
medallion -h
usage: medallion [-h] [--host HOST] [--port PORT] [--debug-mode]
[--log-level {DEBUG,INFO,WARN,ERROR,CRITICAL}]
CONFIG_PATH
medallion v3.0.0
positional arguments:
CONFIG_PATH The location of the JSON configuration file to use.
options:
-h, --help show this help message and exit
--host HOST The host to listen on.
--port PORT The port of the web server.
--debug-mode If set, start application in debug mode.
--log-level {DEBUG,INFO,WARN,ERROR,CRITICAL}
The logging output level for medallion.
If you are wondering why I cloned the cti-taxii-server repository, it is to have the demo config and data files locally which will be used in this tutorial.
The first thing needed is to define the configurations in the Medallion config file. The config file contains:
- configuration information for the backend plugin (including an initial data file)
- a simple user name/password dictionary
Two back-end plugins are provided with medallion,
- The Memory back-end: persists data “in memory”. It is initialized using a json file that contains TAXII data and metadata. It is possible to save the current state of the in memory store, but this back-end is really intended only for testing purposes
- The MongoDB backend is somewhat more robust and makes use of a MongoDB server, installed independently.
For simplicity in this tutorial, I will demonstrate with the Memory back-end.
Older versions of the repository also ship with a sample data file containing STIX 2.1 object under /medallion/test/data/default_data.json that will fill the TAXII server with initial data (you can of course add more / remove it once the server is running, as shown in the last post).
In this sample data file you will see the Discovery URL config (/discovery), API Roots config (e.g. api1, trustgroup1), and the collections config within the API Roots (and the STIX 2.1 Objects added to them).
If it is unclear how these are structured in the file at first glance it will become much clearer as I continue.
Park that for now, lets look at authenticating to the server first.
As required by the TAXII specification, Medallion supports HTTP Basic authorization. The user names and passwords for this are stored in the config file in plain text (remember, it is not production ready).
To add a user on our server, first create a new config file;
vi config_file.json
Then add the following content to add two users;
{
"backend": {
"module": "medallion.backends.memory_backend",
"module_class": "MemoryBackend",
"filename": "medallion/test/data/default_data.json"
},
"users": {
"admin": "Password0",
"user1": "Password1"
},
"taxii": {
"max_page_size": 100
}
}
Now the basic config file is complete, I can run Medallion like so;
medallion config_file.json --host localhost --port 5000
If successful, you should see the server start;
* Serving Flask app 'medallion'
* Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
* Running on http://localhost:5000
Now that the server is running, I can now interact with it.
The following flows demonstrates the logic of how TAXII Clients generally interact with a TAXII Servers API to get STIX 2.1 structured cyber threat intelligence…
TAXII Discovery
It can be assumed a client trying access a TAXII server is unaware of the collections of data that it holds.
That’s where the server discovery is useful. The HOST/taxii2/ should always return the available API roots which show where the data is.
However, if you go to http://localhost:5000/taxii2 in the browser (and enter the credentials, admin:Password0) you’ll see the error;
{
"description": "Media type in the Accept header is invalid or not found.",
"http_status": "406",
"title": "ProcessingError"
}
That’s because the endpoint is expecting a few things.
First need to encode a user:password to base64 to pass in the Authorization header. You can use one of the users in the config file to do this (e.g. admin:Password0 in base64 = YWRtaW46UGFzc3dvcmQw).
Secondly, under the “Required Headers” section of the TAXII 2 specification for the endpoint /taxii2 you’ll see Accept: application/taxii+json;version=2.1.
So I will pass both of these in the header of my request. In a new terminal window (but keep the Medallion TAXII terminal window open), lets discover our new TAXII server…
curl -X GET "http://localhost:5000/taxii2/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"api_roots": [
"http://localhost:5000/api1/",
"http://localhost:5000/api2/",
"http://localhost:5000/trustgroup1/"
],
"contact": "string containing contact information",
"default": "http://localhost:5000/trustgroup1/",
"description": "This TAXII Server contains a listing of",
"title": "Some TAXII Server"
}
This data all comes from the default_data.json file. See the /discovery config;
{
"/discovery": {
"title": "Some TAXII Server",
"description": "This TAXII Server contains a listing of",
"contact": "string containing contact information",
"default": "http://localhost:5000/trustgroup1/",
"api_roots": [
"http://localhost:5000/api1/",
"http://localhost:5000/api2/",
"http://localhost:5000/trustgroup1/"
]
},
You can see in the API response there are API roots…
TAXII API Roots
In the previous example the server exposed the following API Roots to the user; /api/v1/group1, /api/v1/group2, and /api/v1/group3. There could be even more routes on this server, they are just not accessible to the authenticated user.
API Roots are logical groupings of TAXII Collections (and in the future, Channels). A TAXII server instance can support one or more API Roots.
API Roots offer a way for the owner of a TAXII Server to segment the data between groups of users. A user can have access to zero or more API Roots depending on the user permissions assigned to them on the TAXII server.
For example, a single TAXII Server could host multiple API Roots - one API Root for Collections and Channels used by Sharing Group A and another API Root for Collections and Channels used by Sharing Group B. Different consumers will have access to different groups (aka API Roots)/
To see what is inside an API Root you can use the Get API Root Information endpoint.
The GET request for the endpoint takes the form HOST/<API_ROOT>/.
Lets take a look at the /trustgroup1 API Root;
curl -X GET "http://localhost:5000/trustgroup1/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"description": "A trust group setup for malware researchers",
"max_content_length": 9765625,
"title": "Malware Research Group",
"versions": [
"application/taxii+json;version=2.1"
]
}
Again this data is pulled from the servers memory, loaded from the default_data.json file.
As you can see this endpoint is useful as it describes what type of data is in the root.
Now I know this root is what I want, lets look at the TAXII Collections it holds…
TAXII Collections
A TAXII Collection is a logical grouping of threat intelligence that enables the exchange of information between a TAXII Client and a TAXII Server via a TAXII API in a request-response manner.
For example, I might create a collection for compromised credit card intelligence, another for C2 domains, etc. Ultimately a creator can decide what is in the collection. Many producers simply have one collection (that users can filter using the TAXII API parameters, more on that later).
I can discover the Collection in a root using the collections endpoint like so;
curl -X GET "http://localhost:5000/trustgroup1/collections/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"collections": [
{
"can_read": false,
"can_write": true,
"id": "472c94ae-3113-4e3e-a4dd-a9f4ac7471d4",
"media_types": [
"application/stix+json;version=2.1"
],
"title": "This data collection is for testing querying across collections"
},
{
"can_read": true,
"can_write": true,
"id": "365fed99-08fa-fdcd-a1b3-fb247eb41d01",
"media_types": [
"application/stix+json;version=2.1"
],
"title": "This data collection is for testing adding objects"
},
{
"can_read": true,
"can_write": true,
"description": "This data collection is for collecting high value IOCs", "id": "91a7b528-80eb-42ed-a74d-c6fbd5a26116",
"media_types": [
"application/stix+json;version=2.1"
],
"title": "High Value Indicator Collection"
},
{
"can_read": true,
"can_write": false,
"description": "This data collection is for collecting current IOCs",
"id": "52892447-4d7e-4f70-b94d-d7f22742ff63",
"media_types": [
"application/stix+json;version=2.1"
],
"title": "Indicators from the past 24-hours"
},
{
"can_read": false,
"can_write": false,
"description": "Non accessible",
"id": "64993447-4d7e-4f70-b94d-d7f33742ee63",
"media_types": [
"application/stix+json;version=2.1"],
"title": "Secret Indicators"
}
]
}
The response shows 5 available Collections to the authenticated user.
Note some are can_read=true only, others I can can_read=true and can_write=true.
Most TAXII servers offer collections that are are only, that is a collection has can_read=true and can_write=false.
However, although more rare, TAXII servers can be used to provide users write access, can_write=true, in order to add their data into a collection. This is often seen in threat sharing communities who use a collection as a pool of all their intelligence.
Also, remember earlier I mentioned TAXII servers could be used to distribute intelligence in standards other than STIX. The media_type property of each collection tells the Client polling it what type of data is held within in.
I can then drill down into an individual Collection using its ID as follows;
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"more":false,
"objects":[
{
"created":"2014-05-08T09:00:00.000Z",
"id":"relationship--2f9a9aa9-108a-4333-83e2-4fb25add0463",
"modified":"2014-05-08T09:00:00.000Z",
"relationship_type":"indicates",
"source_ref":"indicator--cd981c25-8042-4166-8945-51178443bdac",
"spec_version":"2.1",
"target_ref":"malware--c0931cc6-c75e-47e5-9036-78fabc95d4ec",
"type":"relationship"
},
{
"created":"2014-05-08T09:00:00.000Z",
"id":"indicator--cd981c25-8042-4166-8945-51178443bdac",
"indicator_types":[
"file-hash-watchlist"
],
"modified":"2014-05-08T09:00:00.000Z",
"name":"File hash for Poison Ivy variant",
"pattern":"[file:hashes.'SHA-256' = 'ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c']",
"pattern_type":"stix",
"spec_version":"2.1",
"type":"indicator",
"valid_from":"2014-05-08T09:00:00.000000Z"
},
{
"created":"2017-01-20T00:00:00.000Z",
"definition":{
"tlp":"green"
},
"definition_type":"tlp",
"id":"marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da",
"name":"TLP:GREEN",
"spec_version":"2.1",
"type":"marking-definition"
},
{
"created":"2017-01-27T13:49:53.997Z",
"description":"Poison Ivy",
"id":"malware--c0931cc6-c75e-47e5-9036-78fabc95d4ec",
"is_family":true,
"malware_types":[
"remote-access-trojan"
],
"modified":"2017-01-27T13:49:53.997Z",
"name":"Poison Ivy",
"spec_version":"2.1",
"type":"malware"
},
{
"created":"2016-11-03T12:30:59.000Z",
"description":"Accessing this url will infect your machine with malware. This is the last updated indicator",
"id":"indicator--6770298f-0fd8-471a-ab8c-1c658a46574e",
"indicator_types":[
"url-watchlist"
],
"modified":"2017-01-27T13:49:53.935Z",
"name":"Malicious site hosting downloader",
"pattern":"[url:value = 'http://x4z9arb.cn/4712']",
"pattern_type":"stix",
"spec_version":"2.1",
"type":"indicator",
"valid_from":"2016-11-03T12:30:59.000Z"
}
]
}
Note the use of the more property. This is how pagination is implemented in TAXII.
Here, it is more=false because the max_page_size in the config was 100 results, and there are only 5 result here. However, when more pages are present (in this case where more than 100 results exist for the query), more=true.
Let me modify the config_file_pagination.json to demonstrate…
vi config_file_pagination.json
Then add the following content to add two users;
{
"backend": {
"module": "medallion.backends.memory_backend",
"module_class": "MemoryBackend",
"filename": "medallion/test/data/default_data.json"
},
"users": {
"admin": "Password0",
"user1": "Password1"
},
"taxii": {
"max_page_size": 2
}
}
Here I’ve set the max_page_size to 2 objects (note this only applies to STIX object endpoints).
And now restart the TAXII server…
medallion config_file_pagination.json --host localhost --port 5000
And run the same request…
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"more":true,
"next":"96e8fbed-0f9b-48fb-bff3-d9dece725c10",
"objects":[
{
"created":"2014-05-08T09:00:00.000Z",
"id":"relationship--2f9a9aa9-108a-4333-83e2-4fb25add0463",
"modified":"2014-05-08T09:00:00.000Z",
"relationship_type":"indicates",
"source_ref":"indicator--cd981c25-8042-4166-8945-51178443bdac",
"spec_version":"2.1",
"target_ref":"malware--c0931cc6-c75e-47e5-9036-78fabc95d4ec",
"type":"relationship"
},
{
"created":"2014-05-08T09:00:00.000Z",
"id":"indicator--cd981c25-8042-4166-8945-51178443bdac",
"indicator_types":[
"file-hash-watchlist"
],
"modified":"2014-05-08T09:00:00.000Z",
"name":"File hash for Poison Ivy variant",
"pattern":"[file:hashes.'SHA-256' = 'ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c']",
"pattern_type":"stix",
"spec_version":"2.1",
"type":"indicator",
"valid_from":"2014-05-08T09:00:00.000000Z"
}
]
}
Filtering and querying objects
TAXII Collections often contain a lot of data, thus the need for filtering pagination.
The TAXII specification has a range of parameters a TAXII Client can use to filter the objects returned to make the request/response flow more efficient for requirements.
Object manifests
Instead of getting the entire payload of objects in a collection you can instead request the Object Manifests using the Get Object Manifests endpoint.
This is particularly useful to retrieve metadata to decide whether it’s worth retrieving the actual objects.
For example;
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/manifest/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"more":true,
"next":"a5dddad9-cf65-4c76-a6e9-8308169dc753",
"objects":[
{
"date_added":"2014-05-08T09:00:00.000000Z",
"id":"relationship--2f9a9aa9-108a-4333-83e2-4fb25add0463",
"media_type":"application/stix+json;version=2.1",
"version":"2014-05-08T09:00:00.000Z"
},
{
"date_added":"2016-11-01T03:04:05.000000Z",
"id":"indicator--cd981c25-8042-4166-8945-51178443bdac",
"media_type":"application/stix+json;version=2.1",
"version":"2014-05-08T09:00:00.000Z"
}
]
}
By default Objects returned via the Object/manifest endpoints are returned sorted in ascending order by the date_added property (that is, the time the object was added to the TAXII server). Meaning, the earliest object added to the server appears first.
It is also important to point out the difference between date_added and version properties in the returned response of the manifest endpoint.
date_added refers to the time the object was added to the TAXII server. Whereas version equals the modified property of the most recent STIX Object in the Collection.
Pagination
As noted last time I can page through the results, because on my server I have set the max_page_size=2.
If the more property is set to true in the response then the next property should also be populated so that the client can paginate through the remaining records using the next URL parameter along with the same original query options.
Important note, the next value is regenerated each time, it is not static. Try is by running the request above and seeing how the next value is different. This means you should not cache the next value for pagination at a later time.
Let me show you how…
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/manifest/?next=a5dddad9-cf65-4c76-a6e9-8308169dc753" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"more":true,
"next":"4adac81a-3d0e-4c1d-a0a6-fa1431569e64",
"objects":[
{
"date_added":"2017-01-20T00:00:00.000000Z",
"id":"marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da",
"media_type":"application/stix+json;version=2.1",
"version":"2017-01-20T00:00:00.000Z"
},
{
"date_added":"2017-01-27T13:49:59.997000Z",
"id":"malware--c0931cc6-c75e-47e5-9036-78fabc95d4ec",
"media_type":"application/stix+json;version=2.1",
"version":"2017-01-27T13:49:53.997Z"
}
]
}
And again…
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/manifest/?next=4adac81a-3d0e-4c1d-a0a6-fa1431569e64" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"more":false,
"objects":[
{
"date_added":"2017-12-31T13:49:53.935000Z",
"id":"indicator--6770298f-0fd8-471a-ab8c-1c658a46574e",
"media_type":"application/stix+json;version=2.1",
"version":"2017-01-27T13:49:53.935Z"
}
]
}
And we’ve reached the last page. There are 5 results on the server.
Lets increase the max_page_size to 100 again to show other ways to paginate through the results.
If I run the original request again…
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/manifest/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"more":false,
"objects":[
{
"date_added":"2014-05-08T09:00:00.000000Z",
"id":"relationship--2f9a9aa9-108a-4333-83e2-4fb25add0463",
"media_type":"application/stix+json;version=2.1",
"version":"2014-05-08T09:00:00.000Z"
},
{
"date_added":"2016-11-01T03:04:05.000000Z",
"id":"indicator--cd981c25-8042-4166-8945-51178443bdac",
"media_type":"application/stix+json;version=2.1",
"version":"2014-05-08T09:00:00.000Z"
},
{
"date_added":"2017-01-20T00:00:00.000000Z",
"id":"marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da",
"media_type":"application/stix+json;version=2.1",
"version":"2017-01-20T00:00:00.000Z"
},
{
"date_added":"2017-01-27T13:49:59.997000Z",
"id":"malware--c0931cc6-c75e-47e5-9036-78fabc95d4ec",
"media_type":"application/stix+json;version=2.1",
"version":"2017-01-27T13:49:53.997Z"
},
{
"date_added":"2017-12-31T13:49:53.935000Z",
"id":"indicator--6770298f-0fd8-471a-ab8c-1c658a46574e",
"media_type":"application/stix+json;version=2.1",
"version":"2017-01-27T13:49:53.935Z"
}
]
}
You’ll see there are no more pages, all 5 object print withing the 100 limit.
If needed you can also limit the results manually on the object/manifest endpoint using the limit parameter, like so;
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/manifest/?limit=2" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"more":true,
"next":"adf37226-e8c0-46f5-9d36-7640ef126b71",
"objects":[
{
"date_added":"2014-05-08T09:00:00.000000Z",
"id":"relationship--2f9a9aa9-108a-4333-83e2-4fb25add0463",
"media_type":"application/stix+json;version=2.1",
"version":"2014-05-08T09:00:00.000Z"
}
]
}
If you’re following along with me you’ll also notice in the header of response two properties; X-TAXII-Date-Added-First and X-TAXII-Date-Added-Last. These headers are returned by the objects/manifest endpoints. These headers contain the date/time value of when the first and last records of those returned on that page (NOT the collection). For example,
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/manifest/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
HTTP/1.1 200 OK
Server: Werkzeug/3.0.1 Python/3.11.6
Date: Thu, 2 Dec 2020 21:11:48 GMT
X-TAXII-Date-Added-First: 2014-05-08T09:00:00.000000Z
X-TAXII-Date-Added-Last: 2017-12-31T13:49:53.935000Z
Content-Type: application/taxii+json;version=2.1
Content-Length: 1008
Connection: close
{
"more":false,
"objects":[
{
"date_added":"2014-05-08T09:00:00.000000Z",
"id":"relationship--2f9a9aa9-108a-4333-83e2-4fb25add0463",
"media_type":"application/stix+json;version=2.1",
"version":"2014-05-08T09:00:00.000Z"
},
...
If the more property is set to true and the next property is empty (because this is not a required property for TAXII servers) then the client may paginate through the remaining records by using the added_after URL parameter with the date/time value from the X-TAXII-Date-Added-Last header along with the same original query options. For example;
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/manifest/?added_after=2017-12-31T13:49:53.935000Z" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{}
Returns no results, but I knew this (because there are only 5 records, and the more property for the last request was false)
Object Versions
To retrieve all object versions, you can use the object version endpoint, here I’m using the second object returned in the previous response;
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/indicator--cd981c25-8042-4166-8945-51178443bdac/versions/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"more": false,
"versions": [
"2014-05-08T09:00:00.000Z"
]
}
Here I can see one object exists (objects are versions by modified date in the STIX object).
Let me add another version of this object to demonstrate what it would look like with multiple versions.
Lets add a new version of the Indicator STIX Object (91a7b528-80eb-42ed-a74d-c6fbd5a26116)…
Adding / Updating Objects
Adding objects can be done using a POST request to the API, this applies to adding new objects and updating existing objects (their is no PUT method in the TAXII specification).
Remember back to the first post… the authenticated user must have can_write permissions for the Collection.
The indicator object I want to update currently looks as follows;
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/indicator--cd981c25-8042-4166-8945-51178443bdac/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"more":false,
"objects":[
{
"created":"2014-05-08T09:00:00.000Z",
"id":"indicator--cd981c25-8042-4166-8945-51178443bdac",
"indicator_types":[
"file-hash-watchlist"
],
"modified":"2014-05-08T09:00:00.000Z",
"name":"File hash for Poison Ivy variant",
"pattern":"[file:hashes.'SHA-256' = 'ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c']",
"pattern_type":"stix",
"spec_version":"2.1",
"type":"indicator",
"valid_from":"2014-05-08T09:00:00.000000Z"
}
]
}
I will update it by changing the name property from File hash for Poison Ivy variant → AN UPDATED NAME.
{
"objects":[
{
"created":"2014-05-08T09:00:00.000Z",
"id":"indicator--cd981c25-8042-4166-8945-51178443bdac",
"indicator_types":[
"file-hash-watchlist"
],
"modified":"2020-01-01T00:00:00.000Z",
"name":"AN UPDATED NAME",
"pattern":"[file:hashes.'SHA-256' = 'ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c']",
"pattern_type":"stix",
"spec_version":"2.1",
"type":"indicator",
"valid_from":"2014-05-08T09:00:00.000000Z"
}
]
}
Here’s the request to do that…
curl -X POST "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1" \
-H "Content-Type: application/taxii+json;version=2.1" \
-d "{\"objects\":[{\"created\":\"2014-05-08T09:00:00.000Z\",\"id\":\"indicator--cd981c25-8042-4166-8945-51178443bdac\",\"indicator_types\":[\"file-hash-watchlist\"],\"modified\":\"2020-01-01T00:00:00.000Z\",\"name\":\"AN UPDATED NAME\",\"pattern\":\"[file:hashes.'SHA-256' = 'ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c']\",\"pattern_type\":\"stix\",\"spec_version\":\"2.1\",\"type\":\"indicator\",\"valid_from\":\"2014-05-08T09:00:00.000000Z\"}]}"
{
"failure_count":0,
"id":"753d0471-d014-4959-910e-7d6ccf836b85",
"pending_count":0,
"request_timestamp":"2020-01-02T13:55:48.399973Z",
"status":"complete",
"success_count":1,
"successes":[
{
"id":"indicator--cd981c25-8042-4166-8945-51178443bdac",
"version":"2020-01-01T00:00:00.000Z"
}
],
"total_count":1
}
Here I can see the one object in the request has been successfully added to the collection.
Note, when adding objects in bulk you can re-check the status of a POST job using the id returned after the original POST request. For example;
{
"failure_count":0,
"failures": [],
"id":"753d0471-d014-4959-910e-7d6ccf836b85",
"pending_count":0,
"pendings": [],
"request_timestamp":"2020-01-02T13:55:48.399973Z",
"status":"complete",
"success_count":1,
"successes":[
{
"id":"indicator--cd981c25-8042-4166-8945-51178443bdac",
"version":"2020-01-01T00:00:00.000Z"
}
],
"total_count":1
}
Here the response is the same, but it’s likely if you’re adding 1000’s of objects you’ll see objects moving from pendings to failures or successes.
Querying the versions endpoint again, I can now see both versions of the Indicator.
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/indicator--cd981c25-8042-4166-8945-51178443bdac/versions/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"more": false,
"versions": [
"2020-01-01T00:00:00.000Z",
"2014-05-08T09:00:00.000Z"]
}
If you remember back to the STIX 2.1 tutorial, I showed how SCOs could not be versioned using the modified property (as they don’t contain one).
So lets look as adding two versions of the same SCO to the collection. Firstly, I’ll add;
{
"objects": [
{
"type": "domain-name",
"spec_version": "2.1",
"id": "domain-name--dd686e37-6889-53bd-8ae1-b1a503452613",
"value": "google.com"
}
]
}
curl -X POST "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1" \
-H "Content-Type: application/taxii+json;version=2.1" \
-d "{\"objects\":[{\"type\":\"domain-name\",\"spec_version\":\"2.1\",\"id\":\"domain-name--dd686e37-6889-53bd-8ae1-b1a503452613\",\"value\":\"google.com\"}]}"
{
"failure_count":0,
"id":"48d2c362-521c-4c94-b33f-5b13f7179393",
"pending_count":0,
"request_timestamp":"2020-01-02T19:40:16.411884Z",
"status":"complete",
"success_count":1,
"successes":[
{
"id":"domain-name--dd686e37-6889-53bd-8ae1-b1a503452613",
"version":"2020-01-02T19:40:16.411884Z"
}
],
"total_count":1
}
Note how the TAXII server automatically assigns a version to the object. The version time matches the time of the import.
Regardless, it’s actually impossible to add another version of an SCO. The TAXII server always assumes an SCO is never updated. Let me show you by adding…
{
"objects": [
{
"type": "domain-name",
"spec_version": "2.1",
"id": "domain-name--dd686e37-6889-53bd-8ae1-b1a503452613",
"value": "google.com",
"resolves_to_refs": [
"ipv4-addr--dc63603e-e634-5357-b239-d4b562bc5445"
]
}
]
}
curl -X POST "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1" \
-H "Content-Type: application/taxii+json;version=2.1" \
-d "{\"objects\":[{\"type\":\"domain-name\",\"spec_version\":\"2.1\",\"id\":\"domain-name--dd686e37-6889-53bd-8ae1-b1a503452613\",\"value\":\"google.com\",\"resolves_to_refs\":[\"ipv4-addr--dc63603e-e634-5357-b239-d4b562bc5445\"]}]}"
{
"failure_count":1,
"failures":[
{
"id":"domain-name--dd686e37-6889-53bd-8ae1-b1a503452613",
"message":"Unable to process object",
"version":"2020-01-02T19:43:28.513386Z"
}
],
"id":"ce85894d-733a-4cf1-85da-54dec1ae23c3",
"pending_count":0,
"request_timestamp":"2020-01-02T19:43:28.513386Z",
"status":"complete",
"success_count":0,
"total_count":1
}
In which case, the original SCO would first need to be deleted, and thus only one version of an SCO can ever exist.
Deleting Objects
The TAXII specification also allows for the deletion of objects in a Collection.
Using the DELETE method (without any parameters) will delete the latest version (by highest modified time) of that object.
For example, lets delete the Indicator that was updated;
curl -X DELETE "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/indicator--cd981c25-8042-4166-8945-51178443bdac/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
Note, this endpoint will return an empty 200 if the original request was successful.
Querying the versions endpoint again, I can now see only the first version of the Indicator still exists.
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/indicator--cd981c25-8042-4166-8945-51178443bdac/versions/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"more": false,
"versions": [
"2014-05-08T09:00:00.000Z"
]
}
Running the delete command on the same Indicator again will delete the object entirely (as only one version remains).
curl -X DELETE "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/indicator--cd981c25-8042-4166-8945-51178443bdac/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
And checking for it again using the versions endpoint…
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/indicator--cd981c25-8042-4166-8945-51178443bdac/versions/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
Returns a 404.
{
"description": "Object 'indicator--cd981c25-8042-4166-8945-51178443bdac' not found.",
"http_status": "404",
"title": "ProcessingError"
}
The delete endpoint also allows for filtering to delete specific versions or spec_versions of an Object, using the match[versions] and match[spec_versions] parameters.
I’ll demonstrate by adding some data to delete…
{
"objects": [
{
"type": "attack-pattern",
"spec_version": "2.1",
"id": "attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5",
"created_by_ref": "identity--d2916708-57b9-5636-8689-62f049e9f727",
"created": "2020-01-01T11:21:07.478851Z",
"modified": "2020-01-01T11:21:07.478851Z",
"name": "Spear Phishing",
"description": "Used for tutorial content",
"object_marking_refs": [
"marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da"
]
},
{
"type": "attack-pattern",
"spec_version": "2.1",
"id": "attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5",
"created_by_ref": "identity--d2916708-57b9-5636-8689-62f049e9f727",
"created": "2020-01-01T11:21:07.478851Z",
"modified": "2020-01-02T11:21:07.478851Z",
"name": "Spear Phishing Updated ONCE",
"description": "Used for tutorial content",
"object_marking_refs": [
"marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da"
]
},
{
"type": "attack-pattern",
"spec_version": "2.1",
"id": "attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5",
"created_by_ref": "identity--d2916708-57b9-5636-8689-62f049e9f727",
"created": "2020-01-01T11:21:07.478851Z",
"modified": "2020-01-03T11:21:07.478851Z",
"name": "Spear Phishing Updated TWICE",
"description": "Used for tutorial content",
"object_marking_refs": [
"marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da"
]
}
]
}
curl -X POST "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1" \
-H "Content-Type: application/taxii+json;version=2.1" \
-d "{\"objects\":[{\"type\":\"attack-pattern\",\"spec_version\":\"2.1\",\"id\":\"attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5\",\"created_by_ref\":\"identity--d2916708-57b9-5636-8689-62f049e9f727\",\"created\":\"2020-01-01T11:21:07.478851Z\",\"modified\":\"2020-01-01T11:21:07.478851Z\",\"name\":\"Spear Phishing\",\"description\":\"Used for tutorial content\",\"object_marking_refs\":[\"marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da\"]},{\"type\":\"attack-pattern\",\"spec_version\":\"2.1\",\"id\":\"attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5\",\"created_by_ref\":\"identity--d2916708-57b9-5636-8689-62f049e9f727\",\"created\":\"2020-01-01T11:21:07.478851Z\",\"modified\":\"2020-01-02T11:21:07.478851Z\",\"name\":\"Spear Phishing Updated ONCE\",\"description\":\"Used for tutorial content\",\"object_marking_refs\":[\"marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da\"]},{\"type\":\"attack-pattern\",\"spec_version\":\"2.1\",\"id\":\"attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5\",\"created_by_ref\":\"identity--d2916708-57b9-5636-8689-62f049e9f727\",\"created\":\"2020-01-01T11:21:07.478851Z\",\"modified\":\"2020-01-03T11:21:07.478851Z\",\"name\":\"Spear Phishing Updated TWICE\",\"description\":\"Used for tutorial content\",\"object_marking_refs\":[\"marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da\"]}]}"
Here we’re adding many versions of the same STIX Attack Pattern object.
{
"failure_count":0,
"id":"2d513dd0-06d1-42be-81b1-da34a48c29d7",
"pending_count":0,
"request_timestamp":"2023-12-19T19:54:04.444328Z",
"status":"complete",
"success_count":3,
"successes":[
{
"id":"attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5",
"version":"2020-01-01T11:21:07.478851Z"
},
{
"id":"attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5",
"version":"2020-01-02T11:21:07.478851Z"
},
{
"id":"attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5",
"version":"2020-01-03T11:21:07.478851Z"
}
],
"total_count":3
}
Now, if I was to run the delete command without any parameters in the request, only the latest version of the object (2020-01-03T11:21:07.478851Z) would be removed.
So instead, lets say I wanted to delete the 2nd version 2020-01-02T11:21:07.478851Z. I could do this using match[versions]=2020-01-02T11:21:07.478851Z.
curl -X DELETE "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5/?match\[versions\]=2020-01-02T11:21:07.478851Z" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
Lets recheck the versions…
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5/versions/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"more": false,
"versions": [
"2020-01-03T11:21:07.478851Z",
"2020-01-01T11:21:07.478851Z"
]
}
I can delete all remaining versions in the same way…
curl -X DELETE "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5/?match\[versions\]=2020-01-01T11:21:07.478851Z,2020-01-03T11:21:07.478851Z" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
Note, multiple values passed to fields that support match values can be passed with a comma ‘,’ which is treated as a logical OR. For example, ?match[versions]=2020-01-01T11:21:07.478851Z,2020-01-03T11:21:07.478851Z.
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5/versions/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"description": "Object 'attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5' not found.",
"http_status": "404",
"title": "ProcessingError"
}
Filtering objects
I’ve shown some examples of filtering to achieve pagination, but all object/manifest endpoints can also be filtered to find specific objects
For example, the following filtering parameters can be used with the Get Objects endpoint;
Used for pagination;
added_after: e.g.2015-01-01T00:00:00.000Zlimit: e.g.100next: e.g.2
Used against object properties;
match[id]: e.g.indicator--29aba82c-5393-42a8-9edb-6a2cb1df070b,indicator--ef0b28e1-308c-4a30-8770-9b4851b260a5match[type]: e.g.indicator,sightingmatch[version]: e.g.first,2018-03-02T01:01:01.123Z,lastmatch[spec_version]: e.g.2.0,2.1
For example;
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/?match\[type\]=indicator" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
Returns 2 indicators…
{
"more":false,
"objects":[
{
"created":"2014-05-08T09:00:00.000Z",
"id":"indicator--cd981c25-8042-4166-8945-51178443bdac",
"indicator_types":[
"file-hash-watchlist"
],
"modified":"2014-05-08T09:00:00.000Z",
"name":"File hash for Poison Ivy variant",
"pattern":"[file:hashes.'SHA-256' = 'ef537f25c895bfa782526529a9b63d97aa631564d5d789c2b765448c8635fb6c']",
"pattern_type":"stix",
"spec_version":"2.1",
"type":"indicator",
"valid_from":"2014-05-08T09:00:00.000000Z"
},
{
"created":"2016-11-03T12:30:59.000Z",
"description":"Accessing this url will infect your machine with malware. This is the last updated indicator",
"id":"indicator--6770298f-0fd8-471a-ab8c-1c658a46574e",
"indicator_types":[
"url-watchlist"
],
"modified":"2017-01-27T13:49:53.935Z",
"name":"Malicious site hosting downloader",
"pattern":"[url:value = 'http://x4z9arb.cn/4712']",
"pattern_type":"stix",
"spec_version":"2.1",
"type":"indicator",
"valid_from":"2016-11-03T12:30:59.000Z"
}
]
}
Remember earlier I added 3 versions of an Attack Pattern (and then deleted them)? Let me add them to the server again, so that I can demonstrate how the get objects endpoints deal with versioning.
curl -X POST "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1" \
-H "Content-Type: application/taxii+json;version=2.1" \
-d "{\"objects\":[{\"type\":\"attack-pattern\",\"spec_version\":\"2.1\",\"id\":\"attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5\",\"created_by_ref\":\"identity--d2916708-57b9-5636-8689-62f049e9f727\",\"created\":\"2020-01-01T11:21:07.478851Z\",\"modified\":\"2020-01-01T11:21:07.478851Z\",\"name\":\"Spear Phishing\",\"description\":\"Used for tutorial content\",\"object_marking_refs\":[\"marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da\"]},{\"type\":\"attack-pattern\",\"spec_version\":\"2.1\",\"id\":\"attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5\",\"created_by_ref\":\"identity--d2916708-57b9-5636-8689-62f049e9f727\",\"created\":\"2020-01-01T11:21:07.478851Z\",\"modified\":\"2020-01-02T11:21:07.478851Z\",\"name\":\"Spear Phishing Updated ONCE\",\"description\":\"Used for tutorial content\",\"object_marking_refs\":[\"marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da\"]},{\"type\":\"attack-pattern\",\"spec_version\":\"2.1\",\"id\":\"attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5\",\"created_by_ref\":\"identity--d2916708-57b9-5636-8689-62f049e9f727\",\"created\":\"2020-01-01T11:21:07.478851Z\",\"modified\":\"2020-01-03T11:21:07.478851Z\",\"name\":\"Spear Phishing Updated TWICE\",\"description\":\"Used for tutorial content\",\"object_marking_refs\":[\"marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da\"]}]}"
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5/versions/" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
{
"more": false,
"versions": [
"2020-01-03T11:21:07.478851Z",
"2020-01-02T11:21:07.478851Z",
"2020-01-01T11:21:07.478851Z"
]
}
And now I’ll filter by type=attack-pattern;
curl -X GET "http://localhost:5000/trustgroup1/collections/91a7b528-80eb-42ed-a74d-c6fbd5a26116/objects/?match\[type\]=attack-pattern" \
-H "Authorization: Basic YWRtaW46UGFzc3dvcmQw" \
-H "Accept: application/taxii+json;version=2.1"
See how only the latest object version 2020-01-03T11:21:07.478851Z is returned.
{
"more":false,
"objects":[
{
"created":"2020-01-01T11:21:07.478851Z",
"created_by_ref":"identity--d2916708-57b9-5636-8689-62f049e9f727",
"description":"Used for tutorial content",
"id":"attack-pattern--6b948b5a-3c09-5365-b48a-da95c3964cb5",
"modified":"2020-01-03T11:21:07.478851Z",
"name":"Spear Phishing Updated TWICE",
"object_marking_refs":[
"marking-definition--34098fce-860f-48ae-8e50-ebd3cc5e41da"
],
"spec_version":"2.1",
"type":"attack-pattern"
}
]
}
Normally this is fine as typically it is only the latest object you want. However, this does highlight the importance of the version object endpoint, as there may be many historic versions of the objects being returned from the objects/manifest endpoints.