I previously discussed Generating Star Schemas from Data Vault, which served as the core foundation of our semantic layer. With a data vault containing all the entities and their relationships, we can generate dimension tables directly from entities (hubs/satellites) and fact tables from events (transactional links).

However, a manual operation is still required. Most events only contain references to their belonging entities. For analysis purposes, indirectly referenced entities often need to be identified and added to the generated fact tables. This is usually handled by ETL or ELT processes. Our semantic layer allows users to navigate the entity relationship graphs to specify the paths to the entities to be included in the generated fact tables. Then the corresponding ELT jobs are created by the system. It works great but adds some mental complexity to users and also creates an opportunity to introduce errors.

Generating Fact Tables with Indirect References

First, let's illustrate the approach with a simplified real-world example. Imagine we want to measure the power usage of manufacturing equipment in the factory. We can use intelligent electricity meters to collect the measurements. The uploaded IoT payload from these meters, which contains the meter's ID and power consumption reading, serves as the event data we will use to create our star schema.

Some business systems manage equipment's asset record (like the correlation between electricity meters and equipment, the location and organization of the equipment, the operator of the equipment, etc.). We have replicated them from business systems to build a data vault containing all the entities and their relationships necessary for the analysis.

In the diagram below, the green paths starting from the event "evt_meter" in the graph must be specified to bring equipment, organization, and equipment type to the generated fact table.

This approach supports incremental data loading and can be triggered by the incoming batches of meter events written into "evt_meter." The joins along these green paths dominate the performance overhead and can be adequately managed. The relationship graph does apply some limitations but also offers several options to eliminate loops and duplications.

What is the Unified Star Schema?

The primary intended use of the Unified Star Schema is to enable self-service BI. As its name suggests, the main idea is to use a single "unified" star schema to serve queries. So, instead of having many fact tables each surrounded by some dimensions, a single Puppini bridge table sits in the middle, surrounded by all dimensions and facts.

The Puppini bridge table has all primary key columns from dimension and fact tables. As for rows, it is the "union" of all dimension and fact tables. Naturally, it is sparse and filled with many null values.

One tweak is that for rows coming from fact tables, any "indirect" foreign keys also need to be filled appropriately. The dash-edged areas in the diagram above are the original fact rows' filled "indirect" foreign keys.

This method addresses several commonly seen problems when using dimensional modeling.

First, fan and chasm traps1. The Unified Star Schema natively prevents both kinds of traps from happening. No data can ever be duplicated because it uses union instead of join.

Second, business users usually do not fully comprehend the details of all schemas. It is challenging to get queries correct, especially with multiple fact tables involved (drill-across). The second advantage of the Unified Star Schema is the increased ease of use. Queries always start from the Puppini bridge table, followed by one-level joins to any needed dimensions and facts. A single fixed query pattern makes it considerably easier for business users. Not only is the query more straightforward to write, but without traps.

Third, data-loss problems. Because all dimension keys are also included as separate rows in the bridge table, aggregation does not lose data, even those without transaction or event data within the queried range.

Finding and Saving Indirect References Automatically

You may have noticed the common operation from the previous two paragraphs: finding and saving indirect references. The Puppini Bridge table got me started on thinking about automating this operation.

Initially, we let users manually specify the green arrows in the first diagram. This is not trivial, though. With dozens of entities and all the relationships among them, users often feel lost in the graph. Most of the time, it became trial and error.

Thanks to our data vault approach, all the necessary information is available to find correlated entities. So, we later automated this manual operation. Instead of manually asking users to choose what entities to include, all reachable entities via any number of links are included by default. Every event table generates a fact table with all usable dimensions filled in.

It's important to note that our system supports all relationship cardinalities, including one-to-one, many-to-one, many-to-many, and one-to-many relationships. Assuming we start from the left, one-to-one and many-to-one relationships work fine since one row on the left matches, at most, one row on the right. No row 'expanding' is possible, i.e., one event row can never become more than one row in the generated fact table.

However, one-to-many and many-to-many relationships may expand rows. For example, a power usage measure of 10 could be duplicated multiple times with a sum of more than 10 (which is, of course, wrong).

We provide an option for these two relationship cardinality to specify the "allocation"2 rule (per-measure level). You specify a percentage for each "n" side to distribute and allocate the measure's amount. Let's see a few use cases:

• An electricity meter is connected to and used by multiple machines. The power usage reported from the meter must be adequately "split" among the connected machines. Otherwise, it would be double-counted. In this case, the percentage for each "n" side should have a sum of 100%.

• A sensor measures the indoor temperature. The reading is used for all equipment placed in the same room. In this case, 100% is set for each "n" side.

This option provides the flexibility to support all kinds of use cases. The measure is multiplied by the allocation percentage for that target's row when generating the fact table. Let's see a many-to-many example:

In the above diagram, meter1's power usage reading is evenly split between equip1 and equip2. This is the case for a single meter used by multiple equipment. As another interesting use case, equip1 uses two meters (potentially of two power sources), so its power usage is equal to the sum of meter1 and meter2 (20 + 5 = 25).

Semantic Layer with Unified Star Schema

Our automated fact table generation is similar to that of the Puppini Bridge table generation. Both find and save indirect foreign keys in fact tables. The difference is that our approach keeps fact tables independent and does not mix them physically.

When I first learned about the Unified Star Schema, one immediate concern was the joining between the (super large) bridge and (large) fact table(s). The bridge table only contains primary keys. It must join the fact table(s) to get to measures. Proper partitioning and fine-tuning of the bridge table seem challenging. One approach is to include measures in the bridge table as well, effectively duplicating all fact tables.34 Another approach is to have multiple physical stage tables, one for each fact table, and then create a view to union all the stage tables as the final bridge.

That is where a semantic layer shines. Implementation details do not matter to users. Internally, we use multiple physical fact tables. The semantic layer knows how to generate correct multi-pass SQL against those tables when doing drill-across queries. No fan or chasm trap exists, and there is no data loss either because the semantic layer knows how to avoid the traps.

Conclusion

The Unified Star Schema is quite interesting. Its novel idea of creating a Puppini Bridge table eliminates some of the most common problems in dimensional modeling and enables a more effortless self-service BI experience.

Inspired by the Puppini Bridge table idea, our star schema generation is fully automated and has complete dimensional references. This shows the beauty of the data vault.

A semantic layer provides the best of both worlds. To users, it shields the internal details and helps avoid the most commonly encountered problems when they directly write queries against tables. Implementation-wise, it can use the most optimized schema without worrying whether its users can use it effectively.

Imagine a million excavators each sending one message every few seconds. That's 10GB every few seconds and a few TB every hour. And we must never delete the data.1 So, we use Apache Hudi to store the data archives. It serves many batch data processing pipelines. These data pipelines have their own data loading patterns. Some analyze certain kinds of excavators. Some only analyze excavators within specific cubic capacity (CC) ranges. Some analyze the excavator activity within specific areas.

But there's one thing in common: they are not fast. Most pipelines run in hours. They read large volumes of data and spend most of the time in the reading of data files.

The problem is that most data pipelines read ALL the data first to filter the needed parts. It's impractical to have duplicated data sets with different partitioning schemes and clustering layouts because of the size and the corresponding huge cost.2

But, we do have many frequently-running data loading and querying jobs. They keep reading the same complete data set over and over again while most only need a partial of it. Is there a better way?

The Storage Model

We partition our excavator data by day. The record key includes the excavator ID and the timestamp.

At first, we employed the bucket hashing clustering strategy3, with the excavator ID as the bucket field. This is suitable for retrieving individual excavator data as we only need to read one bucket (out of 128, our configured bucket number). But this is not so good for loading many excavators matching some criteria. By definition, hashing distributes excavators among the buckets evenly. On average, we could expect that most large data pipelines end up reading all the buckets.

The column stats index4 does not help either. It can only help skip reading files without a match. When most of the files indeed contain some matched excavators, there's nothing the data skipping can do.

Fortunately, Apache Hudi does have a clustering feature that can help.

Data Clustering

Apache Hudi's clustering5 is a data layout optimization technique. It stitches together smaller files and also orders the data by sort key to enable data locality.

Let's say we have a table with 2 columns A and B. We choose (A, B) as the clustering key. The clustering split files by the unit of 4 records. Below is a simplified illustration. The records are sorted first by column A and then by column B.

When we query for "A == 2", we only need to read file 3. But, when we query for "B == 2", we have to read all 4 files. Column stats index does help confirm all 4 files do contain matched data, but since it appears in all 4 files, we still need to read all 4 files.

We can see that the first column A dominates the ordering. When we query by only the non-prefix column B, there's not much ordering to use (hence not much locality). It "spreads" out to all files.

So, what we need is a clustering method that can order the data by 2 or more columns, while at the same time preserving the locality for each column alone.

Space-Filling Curve

A space-filling curve is a way of mapping a multi-dimensional space into a one-dimensional space. It can provide the nice "locality" property we desire.

The idea is that if all matched data is closer together on the "curve," sorting by the curve's ordering can put matched data in just a few clustered files.

Let's use the Z-order curve to get an intuition of how it works.

The diagram above shows the comparison of using a Z-order curve and a normal ordering clustering method. A clustering of split size 4 is used. On the left is the Z-order sorted by (x, y), and the right is the normal ordering by (x, y).

If we query for "y == 7," using Z-order only requires reading 4 files (green boxes) while the normal ordering requires reading 8 files. This shows that the use of the Z-order curve does keep the number of data files read small when a non-first column is used alone for querying.

But, when querying for "x == 2", using normal ordering requires reading only 2 files (blue shaded boxes), while using Z-order requires reading 4 files. We are kind of giving up some of the filtering power of the first column for better selectivity for other non-first columns.

Note that the ordering here is not a strict ordering but a rough one. That is not a problem. The main point of using the space-filling curve is the locality it brings to the table. Combined with the column stats indexing, we can find and read the exact data with great efficiency.

If most of your queries only use a single column (or the composite key's prefix), it's better to use the normal ordering method. But if the majority of queries do use different sets of columns, then the space-filling curve method might be worth a shot. It works better with high cardinality columns. It is data-dependent and requires some experiments to find the one best suited for your situation.

In our case, we use the space-filling curve clustering6 with columns vehicle type and ID, timestamp, latitude, and longitude. It serves most frequently run queries pretty well.

Conclusion

For extremely large volumes of data, it might be impractical to have duplicated data sets. We can use space-filling curve clustering to efficiently serve multiple equally frequent query patterns. All major data lakes, including Iceberg, Hudi, and DeltaLake, support space-filling curve clustering. There is also Hilbert curve providing even better multi-dimensional data locality. A lake can have multiple orders at the same time.

Customer: "I'd like to see the daily average utilization ratio of all equipment globally. And I also need to be able to drill down to individual continents, countries, and districts."

Billy: "Sure. Which timezone should be used?"

Customer: "Good question. Our headquarters is in Eastern Standard Time. Let's use EST, then."

Billy went ahead to add the required metrics shortly.

Customer: "This is perfect. Oh, I forgot one thing. All offices should see this in their local timezones."

Billy: "Which timezones do we have to support?"

Customer: "I'm not sure, as we have quite a few locations. Can we support all the timezones? And I think people on travel also need to see this in the local timezone wherever they are."

Billy: "Okay. I think we can dynamically detect the local timezone, no problem."

Well, it turned out there were many significant problems.

Too Many Timezones

First, the world has around 40 timezones, more than just 24 that most people believe. Many countries and areas use 15, 30, and even 45-minute-off timezones. BTW, a country's timezone can change for various reasons, even adding new timezones to the world.1

Billy created separate pipelines, tables, and columns for each timezone. The more timezones, the higher the runtime cost becomes. It was not a viable solution.

Billy returned to the customer to seek to reduce the total number of timezones to be supported. In the end, he added the timezone management features. The customer can configure the set of timezones to be supported. A lot of complexity was introduced.

Inconsistent Metrics

In the customers' high-level operation meetings, executives found that the metrics on their screens occasionally differed from those on others' screens. This created quite a drama.

It turned out people's dashboards were based on their local timezones. The metrics were different here and there, which caused a lot of confusion.

The customer complained about the accuracy of the data. Soon, the system became untrustworthy.

Proper Way to Handle Timezones

To serve the purpose of effective communication, metrics must be consistent. Within a company, it is simply wrong to have personalized dashboards or reports for different time zones.

No matter where the viewer is around the world, the dashboard should always show the same values for the same metrics.

So, how do we handle timezone differences?

The answer to that is we don't. For analytics and reporting purposes, where the viewer is shouldn't matter. What matters is the timezone of the target objects. In this case, the location of the equipment.

For equipment in EST, we use EST to calculate their metrics. For equipment in PST, we use PST to calculate. The equipment utilization ratio would read "85% on Jan 20th" for daily metrics and "83% for 9:00 AM on Jan 20th" for hourly metrics. When we mix and compare equipment from both locations, we only care about "Jan 20th" for daily and "9:00 AM on Jan 20th" for hourly. For business reporting purposes, we don't compare based on the absolute same time (like comparing 9:00 AM EST to 6:00 AM PST).

Technically, the implementation is quite simple:

• Store all timestamps in UTC, along with the original timezone information. Ex. "2024-01-20T09:00:00-05:00" to "2024-01-20T14:00:00+00:00" and "EST", or "2024-01-20T09:00:00-08:00" to 2024-01-20T17:00:00+00:00" and "PST".

• Generate date and hour dimensions based on the original timezone but without the timezone in the output. For example, "2024-01-20" and "09:00" are used for both.

• Use the dimensions to generate the dashboards and reports.

Billy no longer needs separate pipelines, tables, and columns for multiple timezones. And the customer gets a consistent dashboard.

Other Common Timezone Questions

A few other commonly asked timezone-related questions may again cause confusion.

How should we show equipment from multiple timezones together on the same screen?

Viewing the detailed, raw equipment data is more common for monitoring purposes. In this case, show the timezone information along with the timestamp. Make it easy for users to know which timezone individual equipment is in.

Alternatively, some customers ask for conversion to a single timezone on the screen, either detected from the browser's local setting or controlled by users.

Since we store all timestamps in UTC and keep the original timezone information, it is relatively easy to implement either way.

How do we deal with vehicles driving through multiple timezones?

Vehicles should still have an assigned "business" timezone; all reporting should use that as the base. The fact that they drive through multiple timezones does not make it any different. What matters is the elapsed time and the trajectory. Having a base timezone to "attach" the correctly calculated result is all we need.

Conclusion

Always question the requirements and always make sure to talk to the right stakeholders.2 In the case above, the specific customer was from the IT department and was unfamiliar with the business requirements. By unthinkingly following the wrong requirements, the system's complexity and overall cost increase. Worse yet, the system becomes untrustworthy to the actual users.

There is always some late-arriving data. An equipment was disconnected for a few hours and then came back online. Someone forgot to enter a purchase order into the system and found out a month later. A maintenance work order was not marked closed for a few days. It's just a normal part of business.

Late-arriving data handling is messy because there's no boundary on how late it can be. It could be a few seconds, a few minutes, a few hours, a few days, or even a few weeks. Because we don't know when it will come, or even whether there will be any late-arriving data at all, people tend to deliberately set a limit, say 7 days, and just periodically reprocess it for 7 days, just in case there is late-arriving data. That sucks.

I talked about the late-arriving data problem previously and how to deal with it by maintaining two timelines in Data Vault. Now's the time to fill the gap, talking about some practical implementation details.

At a high level, there are two stages of processing required after receiving late-arriving data:

• Correctly saving the late-arriving data

• Properly reprocess its derived data

Let's see each stage in detail.

Saving Late-Arriving Raw Data

With Data Vault, saving late-arriving "raw" data is no different from saving normally-arriving data. We maintain two timelines by also keeping the original business event time, so arriving late does not cause problems. Late-arriving data comes and sits right next to normally-arriving data nicely.

So this is pretty straightforward as it is taken care of by the normal ingestion pipeline and no special handling is required.

We've adopted an incremental loading pattern for Data Vault with our custom dbt materialization. This pattern requires the use of a monotonically increasing timestamp as the basis for incremental data processing. Each incremental batch first finds out from the target table the largest timestamp (essentially the point where the last run finished) and uses it as the lower bound to query the new data to be processed in the current batch. In this way, there's no special bookkeeping needed which makes it very simple.

To use it, simply set the materialization's config timestamp_field. By default, the system-maintained load_datetime is used. It first queries max(load_datetime) from the target table and then uses it as the lower bound to query the source table. load_datetime is system-maintained and is guaranteed to be always monotonically increasing.

{{ config(
  materialized = "rc_incremental",
  timestamp_field = "load_datetime"
) }}

The materialization's config also allows for explicitly specifying the data range by giving the start_timestamp and/or end_timestamp (both optional). If start_timestamp is given, it is used directly instead of querying from the target table. The combination of these 3 configs is useful for one-off data-loading jobs like backloading or incremental initial bulk loading.

{{ config(
  materialized = "rc_incremental",
  timestamp_field = "load_datetime",
  start_timestamp = "...",
  end_timestamp = "..."
) }}

So how is it implemented? In the dbt model, add a replacement stub for the filtering conditions when querying the source table:

{% if model.config.materialized == 'rc_incremental' %}
AND __TIMESTAMP_INCREMENTAL_FILTER__
{% endif %}

The materialization macro replaces the stub with the filtering conditions queried from the target table (and/or start_timestamp/end_timestamp):

{%- set timerange_filter -%}
  {{ dbtvault.build_timestamp_filter(target_relation, timestamp_field, target_timestamp_field) }}
{%- endset -%}

{%- set sql = sql | replace("__TIMESTAMP_INCREMENTAL_FILTER__", timerange_filter) -%}

Refreshing Derived Data

Now that we have late-arriving "raw" data saved into the Data Vault, we still need to refresh any derived data based on it. Otherwise, the derived data would become inconsistent.

There are many kinds of derived data. There could be some business vaults generated from applying business rules (transformation) on the raw vault. There could also be some aggregation data generated from the raw vault for analytics or reporting purposes.

For transformed data, dbt's model dependency with the incremental strategy mentioned in the previous section can take care of it quite nicely. This is no different from handling the normally-arriving data. The same approach also works just fine for non-aggregation facts.

For entities (hub/satellite combos) and relations (link/satellite combos), we need to refresh their PIT tables to include the late-arrived data. To not always regenerate complete PIT tables, the affected data range can be identified by PIT table refreshing jobs. It first uses load_datetime to identify the new data for the current batch, and then from that new data set it identifies the set of effective_datetime (the business time) needed to be refreshed for the current batch. Our dimensions are dynamic views on top of PIT tables. As long as PIT tables are refreshed, dimensions automatically reflect the latest, correct data.

For aggregation facts, we use a different approach.

Not all aggregation methods are commutative like sum or count. Also, deletion (marker) needs to be handled properly. A more general drop-first-then-recreate approach is more suitable. When late-arriving data is received, any aggregation result it logically belongs to is purged first, and then the normal aggregation processing regenerates the result.

Our custom dbt materialization has another config for specifying the target table's column(s) as the replacement key(s). For example, a daily aggregation fact table contains a "date" column which should be set as the replacement key. When some late-arriving yesterday's data is received, it drops yesterday's aggregation results first and then re-generates the aggregation with the latest data (which now includes the late-arriving data).

{{ config(
  materialized = "rc_incremental",
  timestamp_field = "load_datetime",
  replace_by = "date"
) }}

Usually, we use a hashed composite key like hash([date, device_id]) for the replacement key. This way, we can have finer control of the exact data set instead of having to always reprocess everything.

Note that this approach relies on the system-maintained load_datetime to determine the data range to be processed for each batch. This timestamp needs to be centrally generated, either by an orchestrator to assign it while scheduling jobs, or automatically generated by the database server.

Conclusion

Traditionally, late-arriving data handling was messy. Mostly, people ended up using daily jobs handling a rolling n-day window. This was not only slow but also expensive. In Data Vault, accepting and saving late-arriving "raw" data is no different from handling normally-arriving data. They come and get saved correctly together with normal, in-time arriving data. With the same automated, repeatable, and modular pattern, the refreshing of derived data also becomes easy. No more daily job reprocessing the same data over and over again. The reprocessing is launched only when there's late-arriving data, and only for the relevant part. This is fast and cheap. This makes a solid foundation for generating star schema from Data Vault.

The Two Timelines

The first principle of business architecture is never to discard information. Save all business information forever and keep track of all changes. This decouples the data processing from the data ingestion, so we are free to start a new analytical data processing or reprocess an existing one at any time with any slice of history.

In business architecture, there are at least two views: the real-world view and our system’s view. For example, a business contract is signed in the real world, and at a later time, the record of this contract enters our system. There is always some delay between the time the actual event occurs and the time our system records it.

| System Load Date | Contract Num. | Business Date | Amount |
| ---------------- | ------------- | ------------- | ------ |
| 2023-11-02       | 101           | 2023-11-01    | 10,000 |
| 2023-11-16       | 101           | 2023-11-15    | 20,000 |

In the example above, in the real-world contract 101 became effective on 2023–11–01 for an amount of 10K. Later, the amount of it was changed to 20K, effective from 2023–11–15. Since contract data is loaded into the system daily, the system load date is typically on the next day.

The two different worlds of view each have their separate timelines. Do we need to care about both timelines? Well, that depends on your requirements. Generally, there are two categories of questions we have to answer:

• What is contract 101’s amount, right now? (20K)

• What was contract 101’s amount in the system on 2023–11–10? What was it on 2023–11–20? (10K and 20K, respectively)

The first question certainly needs to be answered, as it pertains to actual business. The latter is typically for auditing purposes. If you do need to answer the second kind of question, then you need to maintain the second timeline for the system’s view.

However, even if you don’t have requirements for the second kind of question right now, how can you be sure your business users won’t change their minds later? Since the system timeline is unrecoverable after the initial ingestion process is done, we’d better keep it right from the beginning in any way.

So, how do we maintain two timelines?

For each event entering our system, we need to record both:

• When this event happens in the real world.

• When this event is recorded by our system.

The ability and flexibility we want is to be able to answer both categories of questions above. That’s what we want to enable in the architecture.

Late-Arriving Data

At this point, you might wonder whether we can just use the system’s timeline to represent the real-world timeline. The answer to that comes down to the problem caused by late-arriving data doing retroactive updates.

| System Load Date | Contract Num. | Business Date | Amount |
| ---------------- | ------------- | ------------- | ------ |
| 2023-11-02       | 101           | 2023-11-01    | 10,000 |
| 2023-11-16       | 101           | 2023-11-15    | 20,000 |
| 2023-11-22       | 101           | 2023-11-01    | 15,000 |

Let’s expand the same contract example a bit. On 2023–11–22, a correction was made to change contract 101’s amount to 15K effective from 2023–11–01. This is the so-called late-arriving data doing retroactive updates. The reason might be a correction to human error or for some business reason, but it does not matter. From the real-world view, once the correction was loaded into the system, contract 101’s amount on Nov 1 should be 15K, no longer be 10K.

You can think of the real-world view as the system’s “as-of-now” view. You’ll also notice below the system view as of 2023–11–30 changes because the correction was applied on 2023–11–22.

If we only keep system load dates and treat them as real-world business dates, we will mistakenly identify contract 101’s amount to be 10K from 11–02 to 11–16, 20K from 11–16 to 12–01, and 15K from 12–01 to now. This is not the expectation of correcting the amount to be 15K from 11–02 to 11–16.

Note that many source business systems do not provide real-world business dates, and the best we can get (and approximate) is using our system’s load dates as the business dates. In these cases, this is the best we can do. For other cases, when business dates are indeed available from the source systems, keep and use them.

One more thing about late-arriving data. There might be downstream data processing (directly or indirectly) affected by the late-arriving data, which needs to be identified and reprocessed automatically. All the affected time ranges as well as all the existing results processed with data from those time ranges need to be identified and reprocessed.

Maintaining Two Timelines with Data Vault

Below are the standard table schemas for all our data vault constructs. We standardize not only the schema but also all the table and column names.

The system timeline uses the data vault’s standard load_datetime column. For the real-world business timeline, column effective_datetime is added to the satellite (transactional links do not have associated satellites hence are directly added to the links themselves).

As mentioned in the previous section, whenever the source systems do provide a business timeline, we recommend using that. Otherwise, the best is to use the system timeline to approximate it. If users do not write values to effective_datetime, our system automatically uses the value from load_datetime to write it.

That finishes the normal ingestion flow. What about the late-arriving data mentioned above?

Remember in a data vault, satellite data is compressed in SCD2 structure. Because of that, things get a little tricky because we need to consider the chronologically immediate neighbors, both the previous and the next, to decide what and how to insert into satellites.

Let’s use a simple example. Originally, both T1:(1) and T3:(3) were already processed and entered into the data vault. Since satellites are in SCD2 structure, if T1:(1) and T3:(3) have the same value, only T1:(1) is saved in the satellite. On the other hand, if they have different values, both T1:(1) and T3(3) are saved in the satellite. T2:(2) is the late-arriving data, which is chronically between T1 and T3.

There are 5 possible scenarios to be considered:

1. Originally both T1:(1) and T3:(3) are A, hence only one row in the satellite. The late-arriving T2:(2)=A is the same as both, hence only one row in the satellite at the end.

2. Originally both T1:(1) and T3:(3) are A, hence only one row in the satellite. The late-arriving T2:(2)=B is different from both T1 and T3, hence we have to insert two rows, one for T2:(2)=B and one for T3:(3)=A to correctly save the history in the satellite. There are 3 rows at the end.

3. Originally T1:(1) is A and T3:(3) is C, hence two rows are saved in the satellite. The late-arriving T2:(2)=A is the same as T1, hence there’s no need to insert a new row for T2. There are still two rows at the end.

4. Originally T1:(1) is A and T3:(3) is C, hence two rows are saved in the satellite. The late-arriving T2:(2)=B is different from T1:(1)=A and also different from T3:(3)=C, hence we need to insert a row for T2. There are three rows at the end.

5. Originally T1:(1) is A and T3:(3) is C, hence two rows are saved in the satellite. The late-arriving T2:(2)=C is different from T1:(1)=A but the same as T3:(3)=C. We have to insert a row for T2. There are three rows at the end. Note that T2:(2)=C and T3:(3)=C have the same value and normally should be compressed to leave only one row for T2. However, to “never discard information”, we choose to keep them both.

Patrick Cuba has an awesome article on how to use an extended record-tracking satellite (XTS) to support this processing.

As mentioned in the previous section, after T2:(2) is processed and entered into the satellite, any downstream processing affected by this change needs to be identified and reprocessed.

Querying the Two Timelines

With the two timelines maintained in our data vault, how do we query them?

Remember the two categories of questions we need to answer:

• What is contract 101’s amount, right now?

• What was contract 101’s amount in the system on 2023–11–10? What was it on 2023–11–20?

We let users choose with which to generate dimensions. Users can generate different versions, probably one for the business timeline view and another for the system timeline view.

For business-view dimensions, our system first generates the underlying business-view PIT tables. We use snapshot PIT tables, so we first group by the column effective_datetime, and within each group, we select the one row with the latest load_datetime value. So, for each snapshot date, the PIT table rows all point to the correct satellite row based on the business timeline. Finally, the dimension is created by joining the PIT tables to the base hub and satellite tables, with the column effective_datetime included as the time base.

For system-view dimensions’ PIT tables, the only difference is that we pre-filter the data with snapshot dates on the column load_datetime. This ensures that for each snapshot date, only the system's viewable data is present. Then we follow the same process of generating business-view dimensions.

For facts (transactional links), the process is the same. Therefore, users can choose to use the business-view version to answer the first question and to use the system-view version to answer the second question.

Note that the business-view PIT tables do need to be regenerated for the affected parts, which can be done incrementally whenever there is late-arriving data. However, the system-view PIT tables do not need to be regenerated because load_datetime is always monotonically increasing.

Conclusion

Unlike the TVA, we don’t have ever-increasing multiverse timelines to manage or prune. However, we do have two “sacred” timelines to maintain. The “real-world” timeline serves the business requirements, and keeping the full history of this timeline allows for the retrieval of historical values. The “system” timeline serves the auditing requirements to ensure that retroactive updates do not go rogue. Maintaining the two timelines in a data vault can be easily done with an automated, repeatable, and modular pattern applied. Together with systematic star schema generation from the data vault, it serves as a solid foundation for building analytical solutions. Let the Time-Keepers protect your timeline for you.

Our RootCloud Platform provides an easy way for users to construct their enterprise data vault. With it, our users can manage data vault’s table schema, ingest IoT data, incrementally import and transform IT data from external systems, connect BI tools to it, and build applications directly with its APIs. One special sauce we have is the direct integration of OT and IT data. As long as the “link” between OT and IT is available in the vault, our platform can automatically integrate them together to serve business queries.

There’s one caveat: data vault is not intended to be queried directly. Data vault brings many benefits to the table, however easy and fast querying is never one of those. Its creator Dan Linstedt recommends adding Kimball dimensional modeling data marts as outputs specifically to serve business queries. This approach brings the best of the two modeling techniques together. That’s what our system provides.

The data vault serves as the single source of truth within the enterprise. Users can freely pick required entities, relationships, and transaction/event tables to create a star schema easily. The process is also super lightweight (mostly star schema is just composed of views on top of data vault tables) so that users can create as many star schemas as desired. That enables the agility necessary to serve the always-changing business requirements.

There are many advantages of this architecture:

• New data sources can easily be added and ingested to data vault without impacting the data marts.

• Since all the data is in the data vault, dropping or recreating data marts is super easy.

• Different teams can work together, in parallel, on different star schemas, based on the same data vault. No team is going to be disrupted by other teams.

• High agility is achieved. PoC can be done easily by simply creating a new star schema, and populating data from the data vault. The result is either checked and published or rejected and scraped.

• Business rules are implemented and governed in the business vault and, hence can be shared by all teams.

From Data Vault to Star Schema

So how did we achieve it?

As we use dbt internally for populating the data vault, naturally we also want to use dbt to create and populate data marts. We have created a few dbt macros for generating star schema dimension and fact tables from data vault constructs. This conforms to our automated, repeatable, and modular patterns.

In our data vault, we model business entities as hub and satellite(s) combo, and relationship as link and satellite(s) combo. For transactional events or IoT measurement data, we use transactional links.

In a star schema generated, dimensions are based on entities, and facts are based on transactional links. Relationships are not mapped directly but are used to connect entities for snowflake-style usage or for denormalizing into flattened facts.

For dimension tables, we provide two flavors: with and without history tracking. You get to choose between SCD1 and SCD2 for individual dimensions (of course, our query API handles that for users automatically). In order to efficiently generate dimension tables, we take advantage of point-in-time tables (PIT). By periodically and incrementally generating PIT, our star schema dimension tables can simply be views defined on PITs together with their base hubs/satellites. This is key to our users as they get SCD2 for free, if and when needed.

Based on the awesome AutomateDV, below is the dbt macro for generating a dimension table using PIT:

{%- macro dimension(src_pk, src_nk, src_extra_columns, satellites, exclude_columns, src_ldts, source_model, pit) -%}

    {{- dbtvault.check_required_parameters(src_pk=src_pk, src_nk=src_nk,
                                           src_ldts=src_ldts,
                                           source_model=source_model) -}}

    {{- dbtvault.prepend_generated_by() }}

    {{ adapter.dispatch('dimension', 'dbtvault')(src_pk=src_pk, src_nk=src_nk,
                                                 src_extra_columns=src_extra_columns,
                                                 satellites=satellites,
                                                 exclude_columns=exclude_columns,
                                                 src_ldts=src_ldts,
                                                 source_model=source_model,
                                                 pit=pit) -}}
{%- endmacro -%}

{%- macro default__dimension(src_pk, src_nk, src_extra_columns, satellites, exclude_columns, src_ldts, source_model, pit) -%}

{%- set source_cols = dbtvault.expand_column_list(columns=[src_pk, src_nk, src_extra_columns]) %}
{%- set include_columns = [] %}
{%- set exclude_columns = exclude_columns | map("lower") | list %}
{%- for col in source_cols -%}
    {%- if col | lower not in exclude_columns -%}
        {% do include_columns.append(col) %}
    {%- endif %}
{%- endfor %}

{%- set pit_name = pit['pit_name'] -%}
{%- set pit_pk = pit['pk']['pk'] -%}
{%- set pit_ldts_name = pit['ldts'] %}
{%- set pit_as_of_date_name = pit['as_of_date'] %}

SELECT
    {{ dbtvault.alias_all(include_columns, pit_name | lower ~ '_src') }}
    {%- if pit_as_of_date_name %}
    ,{{ pit_name | lower ~ '_src' }}.as_of_date as {{ pit_as_of_date_name }}
    {%- endif %}
{%- for sat_name in satellites -%}
    {%- if 'extra_columns' in satellites[sat_name] and satellites[sat_name]['extra_columns'] -%},
    {%- set sat_extra_columns = satellites[sat_name]['extra_columns'] %}
    {{ dbtvault.alias_all(sat_extra_columns, sat_name | lower ~ '_src') }}
    {%- endif -%}
{%- endfor %}

FROM {{ ref(pit_name) }} AS {{ pit_name | lower ~ '_src' }}

{%- for sat_name in satellites -%}

{%- set sat_pk_name = (satellites[sat_name]['pk'].keys() | list )[0] -%}
{%- set sat_ldts_name = (satellites[sat_name]['ldts'].keys() | list )[0] -%}
{%- set sat_deleted_name = (satellites[sat_name]['deleted'].keys() | list )[0] -%}
{%- set sat_pk = satellites[sat_name]['pk'][sat_pk_name] -%}
{%- set sat_deleted = satellites[sat_name]['deleted'][sat_deleted_name] %}

LEFT JOIN {{ ref(sat_name) }} AS {{ sat_name | lower ~ '_src' }}
    ON {{ sat_name | lower ~ '_src' }}.{{ src_pk }} = {{ pit_name | lower ~ '_src' }}.{{ pit_pk }}
    AND {{ sat_name | lower ~ '_src' }}.{{ sat_ldts_name }} = {{ pit_name | lower ~ '_src' }}.{{ sat_name ~ '_' ~ pit_ldts_name }}
{% endfor %}

{%- endmacro -%}

and the usage of this macro:

{{ config(
  schema = "mart",
  materialized = "view",
) }}

{%- set yaml_metadata -%}
source_model: hub_ent_equip
src_pk: hk_ent_equip
src_nk:
  - source_column: equip_id
    alias: equip_id
src_extra_columns:
  - source_column: id_pit_ent_equip
    alias: id_ent_equip
pit:
  pit_name: pit_ent_equip
  pk:
    pk: hk_ent_equip
  ldts: load_datetime
  as_of_date: effective_datetime
exclude_columns:
  - hk_ent_equip
src_ldts: load_datetime
{%- endset -%}

{% set metadata_dict = fromyaml(yaml_metadata) %}

{{ dbtvault.dimension(source_model=metadata_dict['source_model'],
                      src_pk=metadata_dict['src_pk'],
                      src_nk=metadata_dict['src_nk'],
                      src_extra_columns=metadata_dict['src_extra_columns'],
                      satellites=metadata_dict['satellites'],
                      exclude_columns=metadata_dict['exclude_columns'],
                      src_ldts=metadata_dict['src_ldts'],
                      pit=metadata_dict['pit']) }}

For fact tables, two options are available. For simple cases, say a transaction line item data persisted in a transactional link, the corresponding fact table is simply incrementally synchronized from the link table. The line items, products, and users referenced in the data all become the fact table’s foreign references to different dimension tables. This is a simple 1-to-1 mapping from data vault to the generated star schema.

For more complicated cases, usually, a chain of links and hubs need to be joined first to enrich the transactional links with the required dimensions. In this case, we allow specifying the “join chain” and the intended dimensions. Essentially, it creates a “denormalized” fact table. An example would be IoT data, which usually comes with just the sensor ID. Of course, we can go with snowflake, but we are really more star guys. By specifying the “join chain” from the sensor ID to its belonging equipment, maybe also further from equipment to the belonging organization, location, and work order, it is very easy to populate a required fact table to serve different business needs.

Remember since the data vault is the source of truth, any time a new business requirement comes in, we can either update an existing star schema suitable for the purpose, or simply drop-recreate one, or just go creating a new star schema. Either way, all other star schemas are not affected.

Unified View of Data

One of the most important things about a data warehouse is a shared understanding of the business entities and the business processes. This is largely served by the data vault. But what about the dimensional data marts generated from it? Wouldn’t a separate set of data marts conflict with that?

The dimension tables we generated are simply views on top of business entities (hubs and their satellites). Since entities are shared and are the single source of truth, all the dimension tables are guaranteed to be consistent even though different users have their own set of dimension view(s). Essentially, all users are using the same shared dimension (data).

As for fact tables, allowing different users to share physical fact tables has many downsides. Any change to the fact tables would affect someone else, that’s disruption we don’t want to have. To enable full agility, different users must have their own, isolated fact tables. Does that conflict with the goal of having a standard understanding of the business? I believe not, as long as the underlying is the shared data vault, isolated fact tables provide the necessary view of data to different presentation and/or visualization level requirements. And that’s the main point of having multiple data marts.

Star Schema Query Performance

There’s one thing worth mentioning. Originally, we used data vault’s hashkey directly in the generated star schemas. It works fine but the performance suffers as the table size increases. This is mostly due to the hashkey’s length (we use SHA256 which ends up as 128-byte long). For non-trivial fact tables, this became an issue both for the increased table size and the slowed-down query performance.

We have chosen to generate an integer sequential ID for data vault constructs, which is used only in the generated data marts (data vault part always uses hashkey). The sequence ID is stored together with the hashkey in hubs. In the generated star schema dimension and fact tables, the hashkey is removed and only the sequence ID is included. This improves the query performance at least by an order of magnitude.

The creation of the sequence ID column can be standardized in the data vault construct dbt SQL file, like the following:

{% set sequence_id = 'id_ent_equip' %}

{{ config(
  schema = "er",
  materialized = "rc_incremental",
  upsert_by = "hk_ent_equip",
  timestamp_field = "load_datetime",
  ignored_dest_columns = sequence_id
) }}

{% call set_post_sql_header(config) %}
  alter table {{ this }} add column if not exists {{ sequence_id }} integer generated always as identity;
  create unique index if not exists {{ sequence_id }}_{{ this.table }}_true_integer on {{ this }} ({{ sequence_id }});
{%- endcall %}

{%- set yaml_metadata -%}
source_model: stg_ent_equip
src_pk: hk_ent_equip
src_nk:
  - equip_id
src_ldts: load_datetime
src_source: record_source
{%- endset -%}

{% set metadata_dict = fromyaml(yaml_metadata) %}

{{ dbtvault.hub(src_pk=metadata_dict["src_pk"],
                src_nk=metadata_dict["src_nk"],
                src_ldts=metadata_dict["src_ldts"],
                src_source=metadata_dict["src_source"],
                source_model=metadata_dict["source_model"]) }}

Conclusion

With an automated, repeatable, and modular pattern, we made data vault’s integration with Kimball dimensional data marts easy. This separation of concerns serves the business pretty well, as multiple teams get to work together, in parallel, on the same data vault. The evolution of the data vault typically does not affect the data marts (that’s what data vault is designed for), and the business reporting and analytics teams can iterate quickly and independently. And that is really the goal of using data vault.

RSS is long dead. Yes, there is still Feedly, but I found myself go there less and less often. One reason is that usually the interesting content ultimately got shared/recommended to me on either Facebook, Twitter, Medium, or Pocket. When I do go to Feedly (I have more than a hundred feeds there), I kind of marking read all the time… Not a good way to kill time.

On the other hand, for a small percentage less popular content (but must-read for me)[1], I do use IFTTT RSS Feed to automatically save new content to my Pocket. It works perfect for me, except … I have to create one applet per feed and I have to manage them manually. I could use some feed aggregator (there are many options available) but most of them do not allow updating the mixed feed once created.

Well, that is a good opportunity (excuse?) for me to play with something new, more specifically:

1. Event-based architecture

2. Serverless Framework with OpenWhisk

Event-based architecture

It is data centric with various events emitted to trigger different actions. In my case, events can be data changes or time-based schedules.

1. The first area is around the feeds database which simply stores my interested feed URLs. Simple wrapper actions are created for adding-to/deleting-from feeds database.

2. The second area is feed crawling. The feed aggregator needs to constantly check each feed to see if any new items are published. A cron action is triggered every hour, iterating through all the feeds registered. It invokes crawl action to find new feed items and write them to items database.

3. The third area is the notification of new feed items. This is triggered by the OpenWhisk Cloudant feed whenever there’s a new item written to items database. I’m taking advantage of IFTTT to trigger the saving to my Pocket.

Event-based flow makes actions feel more like “glues”, which is a good thing. Most actions are small in size and also mostly isolated from others. They only focus on one and only one “thing”. put and delete actions only deal with saving/deleting feed records. cron action only fetches the list of feed URLs from database and pass them to crawl action. crawl action only fetches feeds to parse the items to write them to database. ifttt action is triggered whenever there’s a new item written to items database and only notifies IFTTT with the received feed item URL.

This makes extending the system easy and safe. I later added a web action to return an RSS feed of the latest 20 items. A simple Cloudant query index was created to return the 20 items and the web action simply composes an RSS XML from it. Nothing already there was touched.

This is pretty trivial in functionality, but it does capture the spirit of a serverless architecture.

A few hacks related to Cloudant makes the action code simpler (and more efficient). One is I use feed and feed item URLs used for _id (Cloudant document unique identifier). After a feed is parsed, all the feed items are simply written to items database with a single bulk API. I didn’t bother to check each feed item to see if already existed (that would require 2 API calls per feed item). Since feed item URL is used as _id, Cloudant simply skips those items already exist. After all, I only want one notification for any feed item ever. BTW, the notification from OpenWhisk Cloudant feed only contains _id. Because the need URL is already available, a second GET request to Cloudant is saved.

Another hack is in Cloudant, when documents are deleted, they are only marked so by setting a special field _deleted to be true. If later another document with the same _id is added, the original document would be reused (with _deleted cleared and a new _rev). To avoid duplicate notification, the action only notifies when _rev starts with “1-” (which means it is the very first time added).

One more hack is that crawl action actually does batch processing (multiple feeds at a time). Initially, cron action simply dispatches one feed per crawl action. Most of what crawl action does is waiting for the return of feed retrieval and typically it takes about 2~3 seconds. Batching multiple feeds together for one crawl action invocation does not increase its processing time. Remember, OpenWhisk is billed by actual usage (time and memory). With 20 for the batch size, that becomes 1/20 in cost. That’s a significant cost saving.

You might notice that I use Cloudant npm module directly in my actions, instead of using OpenWhisk’s pre-installed Cloudant package.

Latency is not the reason as my Cloudant usage is at most one level (as I found out previously). The main reason is simply the lack of functionality. For example, the pre-install Cloudant package only supports read/write by document ID. I can’t use bulk operation with it, nor can I use views with it.

The usage of a single cron action to iterate through all feeds is fine for my personal usage. After all, I only got like 10 feeds registered. What if you have a million feeds?

With a batch size of 20, that means 50 thousand crawl action invocations in parallel. I think it should still scale because FaaS is supposed to handle this well.

Okay, we might want to be nicer to the OpenWhisk platform so that it is not crashed or slowed down for our unnecessary workload. Or maybe, we just want to be … more fancy, like using different crawl frequency for different feed. We could actually create one cron per feed, by using per feed trigger/action:

> wsk trigger create feed-aggregator-dev-crawl-trigger-0001 \
  --feed /whisk.system/alarms/alarm \
  -p cron "0 * * * *" \
  -p trigger_payload "{\"url\":\"https://bryantsai.com/feed\"}"

> wsk rule create feed-aggregator-dev-crawl-trigger-0001-rule \
  feed-aggregator-dev-crawl-trigger-0001 \
  feed-aggregator-dev-crawl

This needs to be performed in the put action. When a new feed is registered, a cron trigger and a rule are created. When the trigger is … triggered, the payload would be passed as the parameter to the associated action. delete action needs to delete the corresponding trigger and rule. The benefit is we can now spread the workload across the hour (random minute/second). We can also adjust the schedule based on individual feed’s publishing frequency.

The downside of this, of course, is the extra complexity of managing all these per-feed triggers and rules. It is really cumbersome to set these up using the openwhisk npm module as 3 API calls are needed for both creation and deletion. The error handling is just … unmanageable as manual rollback is necessary when things go wrong in the middle. The other downside is cost as we are now back to one feed per action.

I’m happy with a single cron action.

Serverless Framework with OpenWhisk

Basically, Serverless Framework a tool making it much easier to manage your FaaS “projects”. If you have more than just a few actions, using it saves the trouble of creating lots of shell scripts.

Everything, except code, is declared in one file serverless.yml:

1. Action default parameter

2. Feeds: for now, you can define /whisk.system/alarms, /whisk.system/cloudant, /whisk.system/messaging directly as events for actions and Serverless Framework automatically creates/manages the corresponding OpenWhisk triggers and rules for you.

3. Triggers and their parameters: for other OpenWhisk feeds, you can still define custom triggers and Serverless Framework still can automatically create/manage the corresponding OpenWhisk triggers and rules for you.

4. The association of actions and triggers: this is by defining the actions’ events. Again, Serverless Framework manages the corresponding OpenWhisk rules for you.

I can simply issue the command serverless deploy and everything will be set up appropriately on OpenWhisk. No longer the need to manage all those feeds/triggers/rules individually and manually. Managing all related artifacts has never been easier. Automation is the king.

That’s it! It even supports packaging actions as Node.js modules (see crawl action defined in serverless.yml). There’s really nothing else special about using Serverless Framework with OpenWhisk.

Using Serverless Framework with OpenWhisk is a no-brainer to me.

Note that at this point, I’ve encountered a few issues in using Serverless with OpenWhisk:

1. It does not support the management of package binding yet. So I have to use a shell scripts create-cloudant-binding.sh which needs to be run first.

2. It does not support the new API gateway yet, so the http event does not work properly for me. I have to setup the API on OpenWhisk manually for my web action feed.

3. All action names are prefixed with Serverless Framework project name (crawl becomes feed-aggregator-dev-crawl in OpenWhisk). Rules automatically created are also named based on their associated actions/triggers. This helps a lot to OpenWhisk’s cluttered namespace. Except … custom trigger names are not managed in the same way.

Here’s the project, have fun!

bryantsai/feed-aggregator
Contribute to feed-aggregator development by creating an account on GitHub.github.com

Notes

1. In case you are interested, here are my must-read feeds: 
   https://blog.acolyer.org/feed/
   http://highscalability.com/rss.xml
   http://randsinrepose.com/feed/
   http://stratechery.com/feed/
   http://waitbutwhy.com/feed
   http://steve-yegge.blogspot.com/feeds/posts/default?alt=rss
   https://www.confluent.io/feed/

This is the third part of a series.

1. Is it slower?

2. How to structure functions?

3. How to keep state?

FaaS is stateless, which means it provides neither long-term storage nor short-term memory. If we want to keep “something” across invocations, it has to be put somewhere externally.

Watson IoT Analytics has two “states”. One is rules and triggering actions. The other is for stateful rule processing, the so-called “frequency requirement” on rules.

Let’s look at rules and triggering actions first.

In previous parts, I simply embed them in code. That works fine. But wait! Hard coded data? Shouldn’t we use a database?

That’s not hard to do, so let’s use Cloudant for storing rules and triggering actions first. Both rules and triggering actions are stored as documents in Cloudant. A map-reduce view is created to provide an easy way to get all rules with associated triggering actions embedded.

My revised OpenWhisk action now sends a query to Cloudant to get all rules (together with their associated triggering actions):

I’ve added 3 parameters for Cloudant related configuration. They need to be set as default parameters since my OpenWhisk action is called from Message Hub trigger which knows nothing about it.

> wsk action update iot-analytics-3 iot-analytics-3.js
> wsk action update iot-analytics-3 -p cloudant_username <cloudant account> -p cloudant_password <cloudant password> -p cloudant_db <name of cloudant database>

Now the fun part. Since we are using an external database, is it slower than embedded version?

It seems not much. The 95th percentile duration is 0.64 (embedded) v.s. 0.68 (Cloudant) seconds. That’s close enough to be ignored. However, the 99th percentile duration shows the impact of the remote database query on the slower end (worst cases): 0.74 v.s. 1.02 seconds.

Note that I did use a simple cache[1] to avoid querying Cloudant for each invocation, which might be the reason of comparable latency. Most of the invocations simply used the cached data in memory. As mentioned in part 2, OpenWhisk action instances do get “reused/cached” and this kind of optimization is pretty common (and very useful for performance).

For the record, this already works well enough for my use case.

BTW, Watson IoT Analytics does similar caching. In fact, it does not query database at all. All rules and triggering actions are kept in local cache and we have a notification mechanism to inject changes into running instances. This makes it possible to have extremely short processing time.

Embedded v.s. External Database

Before we move on, I want to drill down on this one further.

It’s probably against common (good) programming practice to embed data in code. But if the network to the database is not fast enough, embedded approach is one of few choices for near real-time IoT use cases.

I’ve been pondering on this for some time. The downside of embedded approach is increased complexity in function deployment management. When functions are deployed, we need some kind of template (fragment replacement) to inject the data into the code. This should be built into the deployment pipeline with the database (where the data stored) as input. We obviously need to track the “base” for every deployed function. Also, we need to track the link between a deployed function and the data injected. When new code is out or when data is changed, we need to replace/upgarde all related deployed functions.

That sounds complicated! But there are a few reasons that this may not be as bad an idea as it seems:

1. Before FaaS, having a general code serving different “data” means we only need to run one or just a few servers. This is called multitenancy.

2. With FaaS, this no longer is necessary. Since FaaS only charges for real usage, we can conveniently deploy one instance per “tenant”. The unit can be per user or even per rule. No use, no cost, no worry.

3. FaaS also takes care of elastic scaling, by individual deployed function. For peak, off-peak and even no traffic period.

4. The data to be injected into one function is small. It is just a single or a few rules of one user.

5. FaaS deployment is both easy and fast. When data is changed, instant redeploying related functions is no big deal.

If the latency of using external database is of concern, this could be a viable option.

That being said, as long as the external database works fine, I will not pursue this approach at least for now.

Stateful Processing

Watson IoT Analytics rules can have frequency requirement specified. It basically is count and/or time-based constraints controlling whether rule triggering is to be carried out.

Stateful processing requires short-term storage for storing intermediate results. The fastest short-term storage can be in-process memory or host storage. Unfortunately, FaaS does not provide that. To implement the “frequency requirement” on rules, we need to use external storage for keeping the “state”.

For example, for the 3rd option “Trigger only the first time conditions are met and reset when conditions are no longer met”, we can save each incoming message‘s timestamp along with the rule match result in Cloudant. Whenever there’s a rule match, we fetch the list of messages sorted by timestamp in descending order. If the immediate previous message did not have a match, we trigger the rule’s associated actions; otherwise, we don’t.

However, there’s another restriction of FaaS (at least the current options available on the market) which makes stateful “streaming” processing difficult. There’s no ordering guarantee on invocations. The following is from OpenWhisk’s documentation:

Invocations of an action are not ordered. If the user invokes an action twice from the command line or the REST API, the second invocation might run before the first. If the actions have side effects, they might be observed in any order.

Additionally, there is no guarantee that actions will execute atomically. Two actions can run concurrently and their side effects can be interleaved. OpenWhisk does not ensure any particular concurrent consistency model for side effects. Any concurrency side effects will be implementation-dependent.

The only way is to roll out our own “buffering/windowing/watermarking”. I doubt that worth the effort. At this point, if message order is required, I’d say use a streaming processing platform.

Notes

1. The simple cache does not consider changes to the rules and/or triggering actions from Cloudant side. To add the cache stale data eviction, we can use OpenWhisk Cloudant trigger to receive notification of such changes and force reload the corresponding OpenWhisk actions accordingly.

This is the second part of a series.

1. Is it slower?

2. How to structure functions?

3. How to keep state?

My first take on implementing Watson IoT Analytics with OpenWhisk was pretty minimum in function. Now I’m going to start iterating. Here’s the event processing flow in Watson IoT Analytics.

An incoming event first goes through the rule evaluation and then a stateful processing. If the event passes through the rule filtering, it then checks if anything needs to be triggered and then triggers it. Part 1 implements just the rule evaluation and webhook posting, all within a single OpenWhisk action.

The next question is, how to structure my function (or functions) for adding those conditional branching logic?

Latency

There are at least 3 options available[1]. To answer the question, I need to first understand the latency of each option.

1. Inline: all functions are put in a single OpenWhisk action.

2. Sequence: each function is an action and an OpenWhisk sequence is used to chain them together.

3. OpenWhisk npm package: each function is an action and a main controller action uses the npm package to invoke and coordinate them.

I tested the 3 options with 2 different setting. The first setting is just one level of function call (for the inline option):

// increment.js
function main(params) {
 params.value = params.value || 0;
 return {value: inc(params.value)};
}

function inc(value) {
 return value + 1;
}

The other is 10 nested level (for the inline option):

// inline-10-level.js
function main(params) {
 params.value = params.value || 0;
 return {value: inc(inc(inc(inc(inc(inc(inc(inc(inc(inc(params.value))))))))))};
}

function inc(value) {
 return value + 1;
}

for sequence option:

// sequence-main.js
function main(params) {
 return params;
}

----

> wsk action update sequence-1-level sequence-main,increment --sequence

> wsk action update sequence-10-level sequence-main,increment,increment,increment,increment,increment,increment,increment,increment,increment,increment --sequence

for npm package option 1 level (it’s too ugly to show the 10 level code…):

// npm-1-level.js
var openwhisk = require('openwhisk');

function main(params) {
 params.value = params.value || 0;
 var p = {name: 'increment', blocking: true, result: true, params: params};
 var ow = openwhisk(); 
 return ow.actions.invoke(p).then(result => {
 return {value: result.response.result.value};
 });
}

Let’s see the comparison result:

No surprise here. Inline option being the fastest, sequence next, and npm package the worst. The sequence one has bigger variance though.

• As expected, inline option does not differ much in latency (1 v.s. 10 levels).

• It’s understandable each action in a sequence adds some overhead. But it does add up. Also, the more actions in a sequence, the bigger variance it has.

• OpenWhisk npm package uses OpenWhisk REST API to invoke actions, so each action invocation incurs overhead like network and authentication. It adds quite some latency.

Note that the overhead in latency for both sequence and npm package options are fixed (per action). They are not proportional to action duration[2]. My action is short and fast (a few milliseconds) so the overhead seems especially severe. But if your action is on the seconds level, it probably is not a big issue as long as you use just a few nested levels.

What does this tell us? Unless your actions are all shorter than 10 ms, you don’t need to worry much about the latency overhead introduced by using either sequence or the npm package. Using sequence is faster than using npm package, but as I’ll show later, sometimes we have no choice because OpenWhisk sequence is limited (like you can’t short circuit in the middle, or you can only have one flow).

On the other hand, this does not suggest to blindly transform existing code to FaaS world, like one-to-one function to action mapping. That would be very very wrong. The unit of “function” should be anything related to the exposed “service”. There’s no point in further breaking down that unit on FaaS.

Artifact Size

Out of curiosity, I wonder whether the artifact size would affect actions’ performance.

I tried using a 14MB zipped nodejs action. The normal duration stays pretty much the same, but those periodic spikes differ. For this 14MB size the spikes are around 2000 ms, compared to around 50 ms for the original version.

Since the spikes happen pretty consistent (once every 50 runs), it’s not entirely correct to say they don’t affect runtime performance. If you have a large artifact size, you should expect to have 2% chance to hit a slow run[3].

Iteration #2

I’ve made some changes to my action in part 1:

1. I’ve split the webhook posting into its own action. This is because the webhook posting is intended to be a general ”service” which can be used somewhere else.

2. If no event exceeds our threshold, no invocation is made to the separate webhook action. No unnecessary cost.

3. BTW, the webhook action can accept batch of events. Cost saving again.

4. Webhook body can now be customized, similar to Watson IoT Analytics.

5. Rule/webhook metadata are extracted out of the code, which opens the door for being passed in as action parameters.

Note that the latency of this version does not differ from last one.

Notes

1. Another option is to use Message Hub topics as queues between actions. Each action publish it outcome for downstream actions to consume. This option is considerably more cumbersome to setup and obviously not fast. I consider it more of external asynchronous integration.

2. I added a sleep inside my action to make it longer (a few seconds). In that case, the latency overhead stays the same as without the sleep introduced.

3. During the latency tests, I noticed one thing very interesting. Normally my function’s duration is around 3 ms, but it jumps to around 50 ms once every 50 invocations. I tried increasing the invocation rate from 1 event per 200 ms to 1 event per 1 ms. Both showed the same behavior. I guess some kind of caching is based on counts?

Originally published at bryantsai.com on April 26, 2017.

As I explored the serverless world (the FaaS area), I always wondered how it compared to the “normal” way. Would it slow things down? Is it really more cost-effective? What would be the right scope of a “function”? Would use it for stateful processing still perform?

This is the first part of a series as I tried to figure things out.

1. Is it slower?

2. How to structure functions?

3. How to keep state?

IoT analytics fits the event-driven pattern pretty well. A device sends in an event to trigger some analytics processing which runs on a FaaS platform. We have already built Watson IoT Analytics, so it’s perfect for me to compare.

Watson IoT Analytics has a lot of horsepower in the backend, the “servers”, so of course, it scales pretty well, both in the number of users and in the event throughput it is capable of handling. I want to implement the same analytics flow with OpenWhisk, without setting up my own servers. The “serverless” way.

To follow along, some preparation work:

1. Have your Bluemix Watson IoT Platform, Message Hub, and OpenWhisk instances ready.

2. Connect a historical data storage extension on Watson IoT Platform to the Message Hub instance you want to use. For simplicity, I use a default topic all-devices for everything.

3. On Watson IoT Platform, create the device type iot and one device under it named thermal.

Let’s verify if everything is correctly setup. Use Message Hub REST API to create a consumer instance, then use Watson IoT Platform REST API to publish an event. The event should be forwarded to the configured Message Hub topic all-devices. Finally, use Message Hub REST API to fetch the message back from the topic.

> curl -X POST \
  -H "Content-Type: application/vnd.kafka.v1+json" \
  -H "X-Auth-Token: <Message Hub api_key>" \
  --data '{"name": "my_consumer_instance", "format": "binary", "auto.offset.reset": "largest"}' \
https://kafka-rest-prod01.messagehub.services.us-south.bluemix.net/consumers/my_json_consumer

> curl -u "use-token-auth:<device auth token>" \
  -H Content-Type:application/json \
  --data-ascii "{\"temperature\":20}" \ https://<orgId>.messaging.internetofthings.ibmcloud.com/api/v0002/device/types/iot/devices/thermal/events/status

(A few seconds later ...)

> curl -X GET \
  -H "Accept: application/vnd.kafka.binary.v1+json" \
  -H "X-Auth-Token: <Message Hub api_key>" \
      https://kafka-rest-prod01.messagehub.services.us-south.bluemix.net/consumers/my_json_consumer/instances/my_consumer_instance/topics/all-devices
[{"key":"eyJvJ0=","value":"eyJIn0=","partition":0,"offset":4}]

> curl -X DELETE \
  -H "X-Auth-Token: <Message Hub api_key>" \
https://kafka-rest-prod01.messagehub.services.us-south.bluemix.net/consumers/my_json_consumer/instances/my_consumer_instance

Note the forwarding to Message Hub could take a few seconds because Watson IoT Platform batches the events. My experiments told me the batch duration is likely around 6–7 seconds.

Now that our events can reach Message Hub, we can create an action on OpenWhisk to apply analytics rules. What the action does is to forward an event to a webhook endpoint if it’s temperature exceeds some threshold value. This is exactly what Watson IoT Analytics does, at least its most basic function.

The action code is super easy. Since OpenWhisk’s Message Hub feed does batch messages, we have to handle batches in our action. Once we have the code ready, the remaining work is to create an action with it and set it up with a trigger and a rule:

> wsk action create iotp-<orgId> ./iot-analytics.js
ok: created action iotp-<orgId>

> wsk trigger create iotp-<orgId>-trigger --feed /_/Bluemix_serverless-iot-kafka_Credentials-1/messageHubFeed \
  --param isJSONData true \
  --param topic all-devices
ok: invoked /_/Bluemix_serverless-iot-kafka_Credentials-1/messageHubFeed with id 1458846b661c45789007a833cc819621
(... omitted ...)
ok: created trigger iotp-<orgId>-trigger

> wsk rule create iotp-<orgId>-rule iotp-<orgId>-trigger iotp-<orgId>
ok: created rule iotp-<orgId>-rule

I use WaitHook (which is super cool!) as the webhook end point so I can easily verify the result. Send some events again and you shall see them pop up on WaitHook page.

Not too bad, right? A few steps and we have our serverless IoT Analytics!

Latency

So back to my first question: is there any performance penalty for using FaaS?

To understand the latency of this solution, I tested a single device sending events with different rates: from 1 event per second to 1 event per 50 ms[4]. The OpenWhisk action posts to WaitHook in the end. I also open a websocket connection to WaitHook to stream back the response in order to measure the real end to end response time.

The end to end response time (1 through 4 marked in the chart below) is around 5.9 sec (95%) and 3.6 sec (median). This does not seem like fast at all. However, if we look at the OpenWhisk action’s latency, which is from 3 to 4 below, it is only 0.54 sec (95%) and 0.3 sec (median). This actually is quite good because it includes the webhook posting and the websocket communication back to my device, not just the action’s invocation. BTW, it’s 0.23 sec (95%) and 0.21 sec (median) from 1 to 2 below.

So the major time spent is from 2 to 3 (more precisely, it’s the first part from Watson IoT to Message Hub). As I mentioned before, Watson IoT Message Hub extension seems to use a long batch duration, which is probably 6–7 sec. Unfortunately, there’s no tweaking for that available yet.

If this level of latency is acceptable for you, then it is perfect. But if you are looking for a near real-time IoT analytics, this is clearly not going to meet your needs. There’s really no easy “serverless” way[1] unless Watson IoT somehow provides shorter batch duration.

To see how fast it can be, even though not really “serverless”, we can use Watson IoT MQTT feed[2]. This feed allows OpenWhisk actions to directly consume device events from Watson IoT without the intermediate Message Hub. However there’s an intermediate feed provider needs to be deployed. The main difference is, there’s no long batch.

Now the 95th percentile end to end response time is just 0.56 sec (95%) and 0.35 sec (median)! It is much much faster.

With this level of latency, I believe this is good enough for most IoT real-time analytics use cases. Of course, the downside is there’s an extra “server” to maintain/operate/scale.

Scalability

Obviously, it scales without me doing any work. Each incoming event triggers an action invocation on OpenWhisk. The more events or the faster they come, the more action instances are launched by OpenWhisk in parallel. The upper bound is only the limit imposed by OpenWhisk.

Better yet, when there’s no event, no resource is used. No usage, no cost!

Cost

Talking about cost. OpenWhisk charges by actual usage of time and memory. Both of my actions above generally take 300 ms. Unless your action is way faster than 100 millisecond (it rounds up to the next 100 ms), there’s really no need to think too much[3]. For OpenWhisk, just make sure you choose the minimum memory actually required by your action.

One aspect is worth considering though. The cost model is by action invocation. If we compare the two methods I use before. The Message Hub one roughly invokes my action once per 6 seconds (because Wastson IoT batches messages). The MQTT way on the other hand invokes my action once for each incoming message. Let’s say we send in 100 messages in a second, the difference in cost is huge!

This is obviously a tradeoff. I don’t want to have as high as 6~7 seconds latency, but I don’t need invocation per message either (too expensive). Also from the action’s processing time perspective, there’s not much difference between processing 1 or 10 messages at a time. So a better approach would be to enable batching for the MQTT feed.

Development Experience

As you can see, my OpenWhisk action code is so simple that I don’t have much to talk about. But considering how less I did and spent … I have to say I’m totally convinced this is the way to prototype.

One obvious and very enjoyable experience about developing OpenWhisk is: the deployment is damn fast! I usually just issue wsk action update ... right after finishing my code change, then immediately start sending. It ALWAYS works as I expect. I haven’t yet encountered any problem on deployment yet.

One more thing about deployment. An action’s very first invocation takes longer than subsequent invocations. I use a bare minimum action to test, the first invocation (after update) usually takes 50 ms while subsequent takes just 3ms. I guess that’s “warming-up” time and OpenWhisk does some kind of “caching” for action instances. This is great for optimization opportunity.

Like Bluemix and Cloud Foundry, logging is not very convenient. But for this little experiment, command wsk activation poll suffices.

Notes

1. Well, there is one actually: Watson IoT Analytics. For the curious mind, the end to end response time for using Watson IoT Analytics itself is 1.93 sec (95%) and 1.43 sec (median). This is mainly because of Spark Streaming’s micro-batch interval we use.

2. The original version was outdated so I updated it to work with current OpenWhisk. Also, I have encountered performance issue which turned out to be caused by the free Cloudant instance rate limit. I added a simple cache to the feed so that it does not go out to lookup Cloudant for each message.

3. If your action takes much less than 100 ms, then you should look for opportunity to aggregate multiple calls.

4. For the Message Hub approach, I did later test 1 event per 25 ms and 1 event per 10 ms, which amounts to 240 and 600 events per batch per invocation. Understandably, the processing time of each invocation takes longer since now it needs to wait the completion of all 240/600 webhook posting. However, the processing time just slightly increases to 6~7 seconds for 25 ms case and ~14 (though with much bigger variance) seconds for 10 ms case .

Watson IoT Analytics rule triggering supports several action types: email, webhook, Node-RED, and IFTTT. In most cases, these actions allow you to integrate with almost any system. You can push a notification to mobile devices, post to a Slack channel, tweet it, persist to some data store, or even issue commands back to the devices.[4]

But sometimes, we just need more. Maybe you don’t have an endpoint for receiving the notification. Maybe your endpoint is behind an enterprise firewall. Or, maybe you need some pre-processing applied before you receive the notification.

Many users use Node-RED for these kinds of use cases. The downside is an extra server set up and its maintenance. Also, the scaling of it has always been a challenge.

What if we could use any custom code as actions for Watson IoT Analytics, without setting up a server? That would be nice!

Let’s say we want to publish the notification from Watson IoT Analytics to a message queue so that we can easily decouple it from all downstream applications.

We are going to use OpenWhisk for custom code and Message Hub for the message queue. Since OpenWhisk already has a built-in Message Hub package, we can even avoid writing our own code for this simple use case. First, create a package binding from /whisk.system/messaging with your Message Hub credentials parameters.

> wsk package bind /whisk.system/messaging myMessageHub \
    -p kafka_brokers_sasl "[\"kafka01-prod01.messagehub.services.us-south.bluemix.net:9093\", \"kafka02-prod01.messagehub.services.us-south.bluemix.net:9093\", \"kafka03-prod01.messagehub.services.us-south.bluemix.net:9093\", \"kafka04-prod01.messagehub.services.us-south.bluemix.net:9093\", \"kafka05-prod01.messagehub.services.us-south.bluemix.net:9093\"]" \
    -p kafka_admin_url https://kafka-admin-prod01.messagehub.services.us-south.bluemix.net:443 \
    -p user <Message Hub User> \
    -p password <Message Hub Password> \
    -p topic myTopic

In case you are not familiar with OpenWhisk yet, package binding is just an alias for a package with some default parameters. By using package myMessageHub instead of /whisk.system/message, I don’t need to specify my Message Hub credentials over and over again. As you can see later, it also decouples the usage of this OpenWhisk action from the underlying Message Hub.

With this ready, you can test publishing some messages by invoking the action:

> wsk action invoke myMessageHub/messageHubProduce -p value "This is a test message"

In addition to CLI, OpenWhisk also has REST API for invoking actions. Let’s try it out with curl first:

# first find out your OpenWhisk credentials
> wsk property get --auth
whisk auth
<user>:<password>

> curl -u <user>:<password> -H Content-Type:application/json --data-ascii "{\"value\":\"message body\"}" \
https://openwhisk.ng.bluemix.net/api/v1/namespaces/_/actions/myMessageHub/messageHubProduce?blocking=true

That’s all we needs for Watson IoT Analytics webhook actions.[2]

Now when a rule is triggered, its webhook action would invoke OpenWhisk Message Hub publish action to publish the alert to the topic. The job is done!

Note: webhook action must use application/json content type and the body must contain the key “value” because we use the default Message Hub publish action.

For simplicity, I use OpenWhisk default Message Hub publish action directly. You certainly can write your own action and invoke it in the same way (just replace the package and action name).

What I’m showing here really is about applying any code logic (in any language!) to be triggered from Watson IoT Analytics. With OpenWhisk, you can truly integrate Watson IoT Analytics with any system. The sky’s the limit.

Oh, did I mention there’s no server needs to be setup and maintained? And, it REALLY scales?[3]

Notes:

1. In most cases, this performs pretty well regarding processing throughput. However, if your triggering rate is too high for webhook style processing, it might be better to go with streaming style solution. That is a topic for another day.

2. If it bothers you that webhook body must use a key “value” to wrap the actual message content to be published to Message Hub, you can use your own OpenWhisk action. You need to use a sequence action within which the first action receives and wraps the payload (with the required key “value”), and the second action being the Message Hub publish action.

3. While IBM’s OpenWhisk does scale pretty well, there is currently some limitation set. I believe this will be relaxed and improved over time.

4. There is a recipe showcasing how to stop a device when some condition is met, in the same way by webhook action plus OpenWhisk action.

My project uses Apache Twill and Cassandra Java Driver. Twill uses Guava 13.0.1 while Cassandra Java Driver uses Guava 16.0.1. Even worse, because Twill and Cassandra Java Driver both leak the usage of Guava, my project also directly uses Guava (both versions).

The problem is that Guava 16.0.x is not backward compatible to 13.0.x and it is non-trivial work to upgrade from 13.0.x to it.

This isn’t that rare when you have some most common dependencies like Guava or Apache HttpComponent. Many open source projects use these libraries (of different versions), you ARE going to hit it if you have more than a few dependencies. Usually, the solution is simply to upgrade. We could just patch Twill’s Guava usage to 16.0.x, but Twill is not a small library and its usage of Guava is so fundamental that it takes a lot of effort to go with this option.

One quick and easy way is to shade the dependency. By shading, it means relocating/renaming the dependency to avoid conflicts.

Both Gradle and Maven have plugins for shading. Okay, I can’t believe we’re still using Maven. Anyway, Maven shade plugin can directly relocate classes in addition to building a flat jar. That is while packaging all of a project’s dependencies into a single jar, it can also do optional class relocation if you want.

Here’s how to use it: add the following to Twill parent project POM’s pluginManagement and you get a shaded flat jar.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-shade-plugin</artifactId>
  <version>3.0.0</version>
  <executions>
    <execution>
      <phase>package</phase>
      <goals>
        <goal>shade</goal>
      </goals>
      <configuration>
        <finalName>twill-${project.version}-shaded</finalName>
        <relocations>
          <relocation>
            <pattern>com.google.common</pattern>
            <shadedPattern>shaded.com.google.common</shadedPattern>
          </relocation>
        </relocations>
        <artifactSet>
          <includes>
            <include>*:*</include>
          </includes>
        </artifactSet>
      </configuration>
    </execution>
  </executions>
</plugin>

Note that I relocate all Guava classes to a new package name but NO code change is required. The class relocation is at binary level.

To use this custom built, shaded flat jar, we could use a local repository in our project:

<repositories>
  <repository>
    <id>local-repo</id>
    <url>file://${basedir}/lib</url>
  </repository>
</repositories>

and install the shaded flat jar into this local repository:

mvn \
  org.apache.maven.plugins:maven-install-plugin:2.3.1:install-file \
  -Dfile=./twill-0.6.0-incubating-shaded.jar \
  -DgroupId=org.apache.twill \
  -DartifactId=twill \
  -Dversion=0.6.0-incubating-shaded \
  -Dpackaging=jar \
  -DlocalRepositoryPath=lib

Finally, use the shaded dependency in our project:

<dependency>
  <groupId>org.apache.twill</groupId>
  <artifactId>twill</artifactId>
  <version>0.6.0-incubating-shaded</version>
</dependency>

If you look into the shaded flat jar files, you can see the relocated Guava classes.

# unzip -t lib/twill-0.6.0-incubating-shaded.jar | grep shaded
Archive:  lib/twill-0.6.0-incubating-shaded.jar
    testing: shaded/              OK
    testing: shaded/com/          OK
    testing: shaded/com/google/   OK
    testing: shaded/com/google/common/   OK
    testing: shaded/com/google/common/collect/   OK
    .
    .
    .

Usually, that’s all we need to do. The Twill’s usage of Guava 13.0.1 is now renamed and isolated in that flat jar. But remember I said both Twill and Cassandra Java Driver exposes Guava classes to our code?

Because of this, whenever Guava is used for Twill (like passing Guava-type based objects as arguments), I have to use the package “shaded.com.google.common”. But when Guava is used for Cassandra Java Driver, I still use package “com.google.common”.

This does require code change though, a simple find-n-replace import package name. Not ideal, but it has to be like this.

1. at-most-once: In this mode messages could be dropped if the processing fails or is timed out. This mode requires no special handling and the messages are processed in the order produced by spouts.

2. at-least-once: This mode tracks whether each spout tuple is “fully” processed within a configured timeout. Any input tuple not fully processed within the timeout is re-emitted. This means the same tuple could be processed more than once and messages can be processed out-of-order. This mode does require user code to follow some “rules”, which are briefly described below.

3. exactly-once: With Trident, Storm can provide “exactly-once” guarantee. This will not be discussed further in this post (maybe another in the future).

If your application does require “at-least-once” guarantee, your topology code needs to do the following 3 things:

1. When spouts emit tuples, specify a unique message ID. If you use spout implementation, like storm-kafka or storm-kestrel, they takes care of it so you don’t need to worry about it.

2. When bolts emit tuples, anchor them with input tuples.

3. When bolts are done processing the input tuple, ack or fail the input tuple.

That’s it! If anything goes wrong, Storm would re-emit the failed spout tuples. (Actually it is the spout’s responsibility to re-emit tuples when its fail method is called. Luckily in most cases, we don’t have to write our own spout implementation.

So how does Storm implement this? Storm’s implementation is actually quite ingenious. Besides, I believe a little bit understanding of the internals is always a good thing. It helps the grasp of concepts.

I’ll use the following topology as the example, which is composed of 2 spouts and 2 bolts. Also, there are 2 ackers configured. Note that “sid1” and “sid2” are the IDs of the 2 spout tasks.

When a spout emits a tuple, it notifies a specific acker task about the new tuple’s ID and its own task ID. Because there could be multiple acker tasks, a simple mod hash function is used to determine which acker task to notify. In this case, spout task “sid1” emits tuple “stid1” and notifies Acker1 (mod-hash(stid1)=Acker1) with “[stid1, sid1]”. Upon receiving this info, Acker1 would create an entry for “stid1”. The bookkeeping entry contains 2 information: the originating spout task ID “sid1” and a so-called “ack val”. The “ack val” is initially set to be the spout tuple’s ID “stid1”, which is “1010” in binary form (we use 4-bit value for simplicity in this example. In reality, Storm uses 64-bit value). The same happens for the other spout’s generated tuple “stid2” (which is handled by Acker2).

Bolt1 receives tuples “stid1” and “stid2” and then emits tuple “tid1” anchored to both (multi-anchoring). Since Bolt1 knows the IDs of both spout tuples received, it can easily determine the correct acker tasks to notify (using the mod-hash function). There are two input tuples, so Bolt1 needs to notify twice, one for each. For “stid1”, it notifies Acker1 with “[stid1, tid1]”. For “stid2”, it notifies Acker2 with “[stid2, tid1]”. Acker1 first looks up the entry for “stid1” to find out the value of its current “ack val”. Then it XOR the current “ack val” with the new tuple’s ID “tid1” and updates the “ack val” with the new XOR result. In this case, 1010 XOR 1100 = 0110. The same process for the “stid2” and Acker2 updates its entry to be 1011 XOR 1100 = 0111.

The use of XOR on all tuple IDs of a DAG (tuple tree) is ingenious. There could be thousands if not ten of thousands of tuples in a DAG and keeping track each individually is neither efficient nor scalable. This method only requires a fixed amount of memory (about 20 bytes per DAG) and is also extremely fast. Also, the XOR strategy does not rely on the ordering of messages received by ackers (see this).

After Bolt1 completes the processing of both input tuples, it ack both and also notifies Acker1 and Acker2 the tuples acked. Acker1 updates the “ack val” of “stid1” to be 0110 XOR 1010 = 1100 and Acker2 updates the “ack val” of “stid2” to be 0111 XOR 1011 = 1100.

Finally, the last Bolt2 in this topology receives tuple “tid1”. When it acks “tid1”, it also needs to notify acker tasks with this input tuple ID “tid1” along with the originating soput tuples’ IDs. Storm always copies the originating spout tuple ID(s) into the emitted tuples so the originating spout tuple ID(s) is always available downstream. In this case, the received tuple “tid1” contains both “stid1” and “stid2”. With spout tuple IDs, Bolt2 can notify both Acker1 and Acker2 the final ack of tuple “tid1”. Both Acker1 and Acker2 then update the “ack val” of “stid1” and “stid2” to be 1100 XOR 1100 = 0000.

At this point, both “ack val” of “stid1” and “stid2” are “0000”. When ackers see a zero “ack val”, it marks the originating spout tuple as completed and calls the ack method of the originating spout task.

And there’s even Fig, which is just awesome!

So why do I want to write about running DB2 on Docker? It turns out there are still some dark areas.

Here it is: https://github.com/bryantsai/db2-docker, after burning several hours looking up here and there.

So what’s difficult about getting DB2 running on Docker?

Docker Storage

DB2 uses O_DIRECT flag which is not supported by the aufs storage backend used by Docker. There are two work-around: changing Docker to use devicemapper storage backend or using volumes. See this thread for more backgrounds.

Requiring change of Docker storage backend is not really an ideal solution, as this is not a per-container option. It’s best to provide an image that anyone can use easily.

So that leaves the option of using volumes. Essentially, volumes bypass aufs storage backend. There are several different ways of using volumes, and I found “Data Volume Container” (some called “data-only-container”) most suitable for DB2 container usage.

In our DB2 container, the complete /home is mounted on a volume from a “data-only-container” whose sole purpose is to export the /home volume. In fact, we don’t even need this “data-only-container” running, we just run it once and the volume exported would be useable by our DB2 container:

# this is how we “create” the data volume
# note the special usage of “true” as the command
> docker run -i — name=db2_data_1 -v /home busybox true

# even it is not running, the volume is usable by 
# other DB2 containers
> docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
fdcb78bdb340 busybox:buildroot-2014.02 “true” 6 hours ago Exited (0) 6 hours ago db2_data_1

# launch a DB2 container using the volume
> docker run — privileged=true -i -t — volumes-from=db2_data_1 — name=db2_inst_1 db2:expc

In this way, the complete DB2 instance data including the database content are stored in that volume, separate from the DB2 container. We could start and stop the DB2 container freely and the database content would still be available.

This of course is much better and much portable than using local file system as the volume. You can also easily backup and restore the volume content if you need.

This is also better than the in-container volume as you get to reuse the same volume across different runs.

Building with Volumes

There’s still one tricky part on using volume for DB2 instance data. When building an image from Dockerfile, there’s no way of using volumes. Yes, there’s a Volume command available in Dockerfile, but that is meant to be mounted when the image being built is actually ran. Not for build time!

In the end, we have to resort to using shell script to build the DB2 instance container in order to have the volume mounted. We actually build a base DB2 image using “compact” installation mode (no instance created) and then use a shell script to run this image with needed volume mounted:

docker run -i — name=db2_data_1 -v /home busybox true

docker run — rm=true -i — volumes-from=db2_data_1 db2:expc /bin/bash <<EOF
 userdel dasusr1;userdel db2fenc1;userdel db2inst1;groupdel dasadm1;groupdel db2fgrp1;groupdel db2grp1
 groupadd db2grp1;groupadd db2fgrp1;groupadd dasadm1;useradd -g db2grp1 -m -d /home/db2inst1 db2inst1 -p db2inst1;useradd -g db2fgrp1 -m -d /home/db2fenc1 db2fenc1 -p db2fenc1;useradd -g dasadm1 -m -d /home/dasusr1 dasusr1 -p dasusr1
 /opt/ibm/db2/V10.5/instance/db2icrt -p $port -u db2fenc1 db2inst1
EOF

When it ends, the DB2 instance is created and persisted in the volume from db2_data_1. We don’t need this “build” container anymore so it is removed right after completion (--rm=true).

Once the DB2 instance is created, we can launch a DB2 container and get into it:

docker run — privileged=true — rm=true -i -t -P — volumes-from=db2_data_1 — name=db2_inst_1 db2:expc /bin/su -c ‘/home/db2inst1/sqllib/adm/db2start;/bin/bash’ — db2inst1

From there, you can create/use databases.

Privileged Mode

You would see the following error when starting DB2 manager if you don’t run the container in privileged mode:

$ db2start
SQL1042C An unexpected system error occurred.

According to https://github.com/jeffbonhag/db2-docker,

DB2 has a problem where it needs more shared memory than Docker originally provides.

That’s why we add --privileged=true parameter when running it.

Bonus Points

I’m using Mac OS X and this is another major contributor of spending so much time on this.

Check out How to Use Docker on OS X: The Missing Guide if you use Docker on Mac OS X. There are at least two take-way.

On Mac OS X, because there’s a “boot2docker” VM serving as the Docker host, you cannot directly mount OS X local files or directories as volume. The normal Docker mount is meant for the Docker host, and on OS X it’s the “boot2docker” VM.

Luckily there’s a convenient solution, check out boot2docker together with VirtualBox Guest Additions. With it, you can mount files and directories under /Users on OS X using Docker command directly.

The other take-way bonus point is to get you inside a running Docker container. What does that mean?

Docker containers are often ran in detached mode. Unless you enable sshd (be cautious and see this), you cannot get into a running containers. Sometimes we just need to get into there. So how can we do that without sshd?

Create the following shell script and put it somewhere in your `PATH`:

#!/bin/bash
set -e

# Check for nsenter. If not found, install it
boot2docker ssh ‘[ -f /var/lib/boot2docker/nsenter ] || docker run — rm -v /var/lib/boot2docker/:/target jpetazzo/nsenter’

# Use bash if no command is specified
args=$@
if [[ $# = 1 ]]; then
 args+=(/bin/bash)
fi

boot2docker ssh -t sudo /var/lib/boot2docker/docker-enter “${args[@]}”

Then you can do some crazy stuff like these:

> docker-enter db2_inst_1 ps -A
 PID TTY TIME CMD
 1 ? 00:00:00 su
 8 ? 00:00:00 bash
 97 ? 00:00:00 db2syscr
 99 ? 00:00:01 db2sysc
 105 ? 00:00:00 db2syscr
 106 ? 00:00:00 db2syscr
 107 ? 00:00:00 db2syscr
 109 ? 00:00:00 db2vend
 117 ? 00:00:00 db2fmp
 118 ? 00:00:00 bash
 489 ? 00:00:00 ps

> docker-enter db2_inst_1
% hostname
f4c1b9530fef

How cool is that!

if defined? MyString && obj.instance_of?(MyString)
 # …
else
 # …
end

It turned out to be operator precedence (&& has higher precedence).

obj = ‘str’

defined? String # => “constant”
defined? String && obj.instance_of?(String) # => “expression”
defined?(String) && obj.instance_of?(String) # => true

# MyString is not defined yet
defined? MyString # => nil
defined? MyString && obj.instance_of?(MyString) # => “expression”
defined?(MyString) && obj.instance_of?(MyString) # => false

The original condition is actually equal to defined?(MyString && obj.instance_of?(MyString)). According to defined?, the expression given to defined? is not executed, which means MyString && obj.instance_of?(MyString) is always an ”expression”.

That’s the price for not memorizing the operator precedence table… However, since defined? has such low precedence, I guess it’s easier to just always add parenthesis.

Encoding in Ruby could drive you crazy if you have not dealt with it before or if you are coming from other languages like Java and Python. I found Yehuda Katz has the most comprehensive explanation on the encoding topic. You should read his Encodings, Unabridged and Ruby 1.9 Encodings: A Primer and the Solution for Rails first. If you ever want to know more background on encoding and Unicode, Joel Spolsky has the best treatment: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!).

It’s actually is not all that hard.

In Ruby (1.9+), each String has its own Encoding. This is different from other languages like Java and Python, which transcodes every String to the same Unicode. With all Strings using the same encoding, Strings can be freely mixed together by various operations (like concatenation). Developers don’t need to worry about encoding when dealing with Strings. Do you remember seeing encoding exception thrown from String operation in Java? This is not the case with Ruby:

incompatible character encodings: ISO-8859–1 and UTF-8 (Encoding::CompatibilityError)

The reason for this error is dead simple: ISO-8859–1 and UTF-8 encodings are not compatible and we just can’t mix them together (unless both contain only ASCII characters).

Note that it is a runtime error when two Strings with incompatible encoding come together. It is quite likely things work perfectly fine until some day it blows.

To prevent this error, we have to make sure never mix Strings of incompatible encoding together. One way is to manually transcode either one to the other’s encoding. String.encode makes it really easy to do that (more on this method’s usage later in this article).

An even easier way is to make Ruby behave like Java and Python, by default transcode all Strings to the same encoding:

Encoding.default_internal = ‘UTF-8’

(If you are still using 1.9, you also need to add magic comment # encoding: utf-8 in the first or second line of every source file. On 2.0 or later, the default source encoding is already UTF-8.)

Ruby, Ruby standard libraries, and most major libraries should already respect this option. But that’s just half of the story. If you are responsible for bringing Strings into Ruby, you have to make sure they are transcoded form the correct encoding and to the intended encoding:

# Encoding.default_internal = ‘UTF-8’
str = File.binread(“…”) # str.encoding is ASCII-8BIT
str.force_encoding(“SHIFT-JIS”).encode!

The force_encoding tells Ruby to start using the given encoding for the String and then encode! re-encodes it from SHIFT-JIS to Encoding.default_internal. Note that force_encoding does not cause any re-encoding but merely re-tagged the String’s encoding. What this code snippet does is reading a file in, using SHIFT-JIS to decode it, then transcoding the read String to UTF-8. You could achieve the same effect with the following code:

# Encoding.default_internal = ‘UTF-8’
str = File.open(“…”, “r:SHIFT-JIS”).read

That’s probably all you need to know about Ruby encoding. You also need to be careful of the source encoding when reading data in (via IO, File, etc.), but that is really not specific to Ruby.

Now, my experience from on dealing with a few encoding related problems.

Dealing with Unknown Encoding

Before you read in a file or receive some data from the internet, you have to know what encoding it uses in order to decode it. What if we just don’t know its encoding?

I’m not sure if there is a complete solution for auto-detecting encoding. At least not that I can find easily.

The first problem from Performance Analysis Suite was with DB2 snapshot text output, which is in plain text without encoding information. Most user databases are now using UTF-8 codeset, so UTF-8 encoding was chosen to read in the snapshots. The only problem with this is with non UTF-8 databases, the SQL queries in snapshots might contain literals incompatible to UTF-8. Either the literal contains illegal byte sequence in UTF-8 or the literal contains some non-ASCII characters which would look weird using UTF-8 encoding.

Since the DB2 snapshot statement tab has simulated statement concentration function, the actual literal values are not really critical to the performance analysis. It is therefore okay for our application to accept some data loss. So we remove all invalid characters from the SQL queries:

str = File.binread(“…”) # str.encoding is ASCII-8BIT
str.encode!(‘UTF-8’, ‘ASCII-8BIT’, invalid: :replace, undef: :replace, replace: ‘’)

Any character from the source encoding ASCII-8BIT that can not be transcoded to UTF-8 would be removed (replaced by empty String). In the end, you get the snapshot text output content read in as UTF-8 String.

Using Binary when You Need Binary

Another problem was related to the dumping of possible binary data to YAML file. Some bitwise operation was applied on each character and then Strings are persisted by YAML.dump. We used to apply bitwise operation directly on the String with UTF-8 encoding. This caused a lot of pain as there’s no reliable way to read it back.

We should have explicitly used force_encoding(“ASCII-8BIT”) before applying bitwise operation:

# change to binary before applying bitwise operation
str.force_encoding(‘ASCII-8BIT’)
result = ‘’.force_encoding(‘ASCII-8BIT’)

# bitwise operation
str.size.times {|i| result << bitwise(str[i].ord, decoding)}

# change back to UTF-8 if decoding
result.force_encoding(‘UTF-8’) if decoding
{% endhighlight %}

When decoding, after bitwise operation we just tag the encoding back to UTF-8.

I’m not here to tell why I don’t like Maven or what’s wrong with Maven. Google around and you’ll easily find many. I just want to share a few tidbits about Maven.

So here’s My typical relationship with Maven:

1. When starting a new project, get an old POM from previous projects.

2. Everything’s fine, great! Leave it alone. Done.

3. Okay, something’s new. Google solutions…

4. Most of the time, there ARE solutions to my new build requirements. But strangely, there are also ALWAYS some tweaking needed.

5. Done. Leave it alone.

6. Until next time…

It’s especially interesting how often the above 3rd and 4th points happen. When using Maven, I found myself constantly need Google (and StackOverflow). That should say something already.

Anyway, I’ve long since learned my lesson (and you probably have already read this somewhere):

Don’t fight Maven

Multi-Module Projects

Maven believes the complete project, including all modules, should be located in a single repository. Modules are put in sub-directories.

“That can’t be right”, you said. “We have to put each module in its own repository.”

It’s a good and valid cry. But usually it ends up spending many hours hacking around.

Remember: don’t fight Maven.

One way is to use Git submodules and to create a new aggregate parent project. Maven would then take care of the module dependencies appropriately. Also, with a little change, you don’t really need an extra aggregate project simply for Maven.

Let’s use a very simple example: a web project A uses jar project B. A and B both are put in their own Git repository. How should we use Maven for them?

We can use a aggregate multi-module POM as the parent which includes both A and B as modules. This aggregate POM is put in A (named pom.xml), as it would be the starting point for A. The POM of the actual A (the web project), named pom_A.xml, is also put in project A along with pom.xml. pom_A.xml is the real deal, responsible for building web project A itself.

Project B is then added as a Git submodule in A, under sub-directory lib/B. In pom_A.xml, we have declared the dependency on B as usual. And that’s it.

<! — pom.xml →
<groupId>…</groupId>
<artifactId>…</artifactId>
<version>1.0.0-SNAPSHOT</version>
<name>…</name>
<packaging>pom</packaging>
<modules>
 <module>./pom_A.xml</module>
 <module>lib/B</module>
</modules>
<build>
 <plugins>
 <plugin>
 <artifactId>maven-clean-plugin</artifactId>
 <version>2.5</version>
 <configuration>
 <skip>true</skip>
 </configuration>
 </plugin>
 </plugins>
</build>
…

<! — pom_A.xml →
<groupId>…</groupId>
<artifactId>A</artifactId>
<version>1.0.0-SNAPSHOT</version>
<name>A</name>
<packaging>war</packaging>
<dependencies>
 <dependency>
 <groupId>…</groupId>
 <artifactId>B</artifactId>
 <version>1.0.0-SNAPSHOT</version>
 </dependency>
…

<! — pom_B.xml →
<groupId>…</groupId>
<artifactId>B</artifactId>
<version>1.0.0-SNAPSHOT</version>
<name>B</name>
<packaging>jar</packaging>
…

One caveat is the skipping of clean phase of the aggregate POM itself. Without this extra configuration, if you use mvn clean package to build, you’d be surprised to see after A and B are built successfully, the build artifacts are cleaned immediately! Like I said, always tweaking…

One more thing is to remember to run git submodule update --init --recursive first. I probably should Google how to include this into Maven as well…

Dependencies Not In Any Maven Repository

How should a private library be handled in a Maven build?

One choice would be to setup an internal Maven repository. If that is overkill, use a local repository in the project and have the private library also checked in.

First install the private library into sub-directory lib (which will serve as the local Maven repository). Add the created repository files/sub-directories into SCM.

Then add to POM an extra repository pointing to sub-directory lib. Declare the usual dependency. Done.

<! — install_lib.sh →
mvn org.apache.maven.plugins:maven-install-plugin:2.3.1:install-file \
 -Dfile=./cglib-nodep.jar \
 -DgroupId=cglib \
 -DartifactId=cglib-nodep \
 -Dversion=2.2.3 \
 -Dpackaging=jar \
 -DlocalRepositoryPath=lib

<! — pom.xml →
<dependencies>
 <dependency>
 <groupId>cglib</groupId>
 <artifactId>cglib-nodep</artifactId>
 <version>2.2.3</version>
 </dependency> 
 …
</dependencies>
<repositories>
 <repository>
 <id>local-repo</id>
 <url>file://${basedir}/lib</url>
 </repository>
</repositories>
…

Browsing through stackoverflow, you could easily find many having the same struggle. We just can’t or don’t have time to figure out all the edge cases ourselves.

So here are the hard and fast rules I figure out and will follow in the future:

Static Resources

Content that never change: JS and CSS files, images, and any kind of binary files all fall in this category.

By never, I really mean never. It’s common best practice to versionize static resources. Whenever they change, so do their URL change.

Here are the simple rules for static resources:

• Embed fingerprint in either the file name or path. Avoid using query string for the fingerprint. Also, ensure the generated URLs differ on more than 8-character boundaries.

• Use these HTTP headers:

Cache-Control: public, max-age=31536000
Expires: (a year from now)
ETag: (based on content)
Last-Modified: (some time in the past)
Vary: Accept-Encoding

It’s that simple for static resources.

Dynamic Resources

Depending on the application’s requirement on freshness and privateness, different cache control setting should be used.

For non-private and constantly changing resources (think of stock ticker), the following could be used:

Cache-Control: public, max-age=0
Expires: (now)
ETag: (based on content)
Last-Modified: (some time in the past)
Vary: Accept-Encoding

The effect is that the resource could be cached publicly (by browsers as well as by proxy servers). Each time before browsers use the resource, they would check whether there’s a newer version and download it if there is.

Note that with this, browsers have some flexibility on revalidation. Typically when users click back/forward buttons, browsers do not revalidate but instead just use the cached version. If you’d like more strict control, say browsers must revalidate even when clicking back/forward buttons, use:

Cache-Control: public, no-cache, no-store

Not all dynamic resources are to become stale right way. If they can be fresh for at least 5 minutes, use:

Cache-Control: public, max-age=300

With this, browsers only revalidate after 5 minutes. Before that, cached content is used directly. If strict control over staleness is also required after 5 minutes, you can add must-revalidate:

Cache-Control: public, max-age=300, must-revalidate

For private or per-user content, replace public with private to avoid the content being cached by proxies:

Cache-Control: private, …

Cache-Control and Expires

When both Cache-Control and Expires are used, Cache-Control takes precedence.

Using both Cache-Control and Expires is meant to gain wider support (by different browsers and versions). Of course, they should be configured to mean the same freshness to avoid any confusion.

See Expires: vs. Cache-Control: max-age.

ETag and Last-Modified

These headers are used when browsers do revalidation. Basically, browsers just blindly store the values of these headers received from the server, and later when validating, browsers send conditional request with these values to the server (via headers If-None-Match and If-Modified-Since, respectively).

Note that validation only occurs after the resource expired.

It is up to the server when both header If-None-Match and If-Modified-Since are present in conditional requests. However, since it is the server generates ETag and/or Last-Modified, in practice, there’s not much problem. Most browsers do send both if available.

See What takes precedence: the ETag or Last-Modified HTTP header?

One frequent suggestion is to avoid the use of ETag. This is not always a valid suggestion. ETag indeed provides more precise control on whether content is really changed. The default Apache method for generating ETag takes file inode, size, and last modified date time as input. This makes the generated ETag value pretty useless in a load balanced environment, because each server will generate a different ETag value for the same file. This is probably the only issue that causes a lot of people to disable ETag completely, which is not really necessary as long as a single unique ETag value is generated for exactly matching file content.

See Should your site be using etags or not?

Manually hitting Ctrl-R

When hitting Ctrl-R, browsers send request with the following headers to check if it needs to refresh the cache content:

Cache-Control: max-age=0
If-None-Match: …
If-Modifed-Since: …

Note that this is not really just talking to the original server, but meant for any proxy servers along the way. Essentially it revalidates the content. If 304 is replied, browser uses the cached content.

Vary: Accept-Encoding

This header might be unfamiliar to some.

When a resource is gzip compression enabled and is cached by proxy servers, clients not supporting gzip compression would get incorrect data (that is, compressed) without this. It instructs the proxy servers to cache two versions of the resource: one compressed, and one uncompressed. The correct version of the resource is delivered based on the request header.

Another reason is the reality: Internet Explorer does not cache any resources that are served with the Vary header and any fields but Accept-Encoding and User-Agent. So adding this header in exactly this way is to ensure these resources are cached by IE.

• Increasing Application Performance with HTTP Cache Headers

• Things Caches Do

• Google Developers: HTTP Caching

• Google Developers: Optimize Caching

• Caching Tutorial for Web Authors and Webmasters

• Cache Control Directives Demystified

• What are the hard and fast rules for Cache Control?

• What’s the difference between Cache-Control: max-age=0 and no-cache?

• HTTP Cache Control max-age, must-revalidate

• Difference between no-cache and must-revalidate

• Caching Improvements in Internet Explorer 9

一樣，怕你看不下去，先來講一下我的結論:

買車險一定要先買第三人意外責任險，而且至少要買到 200 萬以上，有了以後如果還有預算，才去考慮車體險及竊盜險。

這裡的目的不是要省錢，所以我就不弄一堆試算表了，只是一點研究過後的心得報告。

首先，車險有很多細項，至於為什麼要搞成這樣? 你問我，我也不知道要問誰，反正整個就是一個莫名其妙就對了! 不過不管怎樣，要知道賣的是什麼藥膏，還得先搞懂這葫蘆裡到底裝了什麼才行。

強制責任險

這個是政府規定一定得保，而且沒保警察會開罰單的[¹]。這個保險的目的當然是希望所有交通事故的受害者都能受到保障 (不管是開車的、坐車的、甚至是在車子外面不幸受到波及被撞的) 。簡單來說，任何人只要發生交通事故而受傷，就可以到任何一家有承保強制責任險的保險公司申請理賠，即使只是走在路上不幸被別人開車給撞到，或是肇事者的車沒有任何保險，甚至肇事者根本沒有錢來賠你。

既然這是強制性質的，它的保障基本上不會太多[²]，而且它只保人體受傷的部份，並不包括財物損害 (比如說修車費就不行)。它的理賠應該看作是一種最最最基本的保障，一定要再搭配其他的保險才夠，千萬別以為只保了這個就一切搞定 (沒這麼簡單)。

強制責任險的好處在於，嗯 … 它是強制的 … 也就是說每一輛車都得保，因此理論上每一件交通事故的受害人都會受到保障 (至少是跟車子有關的)，事實上也是如此，因為就算肇事者 “好膽” 沒保，甚至是好死不死遇到肇事者逃逸，我們還是可以獲得這部份的理賠 (換句話說，這是政府給我們的一種保障)。關於強制責任險，有一些比較需要注意的項目，比如說在”一車輛事故”[³]中 “駕駛人” 受傷是不理賠的 (好比說新手上路，結果開去撞路燈而受傷)，這部份可能就需要另外保個意外險或是駕駛人傷害險來補強一下。還有，其本上不管我們是受害者還是肇事者，只要有受傷就可以獲得強制險的保障，不過有一些例外情形，像酒駕、無照駕駛等[⁴]，保險公司事後是會向肇事者求償的 (也就是說如果你亂搞，抱歉，請自己去賠對方)。

買這個險沒啥好選的，法律規定一定要保，而且保費也是政府統一定價[⁵]，基本上跟哪一間保險公司買都一樣。強制責任險的保費跟年紀、性別、還有肇事紀錄都有關係 (紀錄是跟著人的)，不過它在車險當中算是相對便宜的，通常二千元上下就差不多了，而且只要一年內沒有肇事紀錄就可以折 18% 的保費，連續兩年沒有則可以折 26%，最高連續三年沒有肇事紀錄可以折 30%。但是反過來說，如果三年內有任何肇事紀錄的話 (紀錄會累計)，最高可能得多付 60% 的強制險保費哦!

第三人意外責任險

強制責任險只保人體受傷的部份，而且保額最高每一位受害人只有 20 萬 (受傷)、150 萬 (死亡或殘廢)，以目前一般的實際案例來講[⁶]，法院判賠都在 300~500 萬之間，強制責任險其實是不夠的。不夠怎麼辦? 不夠就得自己另外付啊! 想想看，出個意外已經夠倒楣了，萬一不幸被判定為肇事者，還得再付出一大筆和解金，付不付得出來還不知道呢，這樣以後的日子要怎麼過下去啊!

這個不夠的差額部份，就有必要買個第三人意外責任險來補強。拿我自己作例子，我是抓一人 450 萬左右，扣掉強制險的 150 萬，還要再補 300 萬的第三人意外責任險才夠。當然，要抓多少看個人，參考一下報紙上的統計數據應該就可以有個底了。

第三人責任險其本上會分為兩個部份，一個是人體的部份，另一個則是財物的部份，是分開保的。這個險的目的很簡單，基本上就是用在賠償對方 (所以叫第三人)，有可能是對方受傷、殘廢、甚至死亡，另一方面則是對方財物上的損失。

這部份的保費真的不貴 (如果跟甲式、乙式比的話)，以我在幾家保險公司詢價的結果來看，100/200[⁷] 大概只要 500 元，提高到 200/400 也不過 850 元左右，再加到 300/600 大概也才 1000 元出頭。至於財損的部份 30 萬大約是 1100 元，提高到 50 萬則是 1400 元左右。換句話說，每年只要多花個大概 2000 元左右 (除了強制險之外再加保第三人責任險)，就可以獲得 “更足夠” 的保障。

很多人願意一年花好幾萬保車體險 (然後在到期前來換個全車烤漆)，卻不願花個幾千塊多買一點責任險，我真的不懂，2000 元真的很貴嗎? 意外嘛，真的是不可預知的，我每次在路上看到一些不知道開車在想什麼的人，就真的覺得就算自己再怎麼小心，也很難保證哪一天不會運氣不好碰到一個白目的人。

記得，第三人責任險一定要買，而且要買夠!

第三人意外責任險 (多倍型)

第三人外外責任險體傷的部份，通常是採用 100/200 的型式 (或著 200/400、300/600，看買多少)，意思是說每一位受害人最高理賠 100 萬，而每一件事故最高理賠到 200 萬。這種雙倍型的很容易會有保障不足的情形發生，比如說對方車上總共有三個人，每個人判賠 85 萬 (總共要賠 255 萬)，這不夠的 55 萬就得自己額外再付。現在 CRV 、休旅車這麼多，萬一碰到對方的車上有四、五個人那怎麼辦?

這幾年保險公司有推出新的 “多倍型” 的第三人意外責任險，可以把單一事故的限額提高，也許是三倍、五倍、甚至是十倍。我自己是選擇五倍 (我的車要撞到可以坐超過五人、而且還把人家撞的很嚴重，機率應該不大)，也就是 300/1500，這樣子至少可以保障到至少五個人的狀況。聽起來很不錯，也這你會覺得這種多倍型的保費一定會貴很多吧!? 不過最讓人訝異的是，以我保的五倍而言，居然只比雙倍型的多不到一百元耶!

記得，問一下你的保險公司有沒有這種商品，沒有就換一家吧![⁸]

超額責任險

這也是一種新的商品，如果覺得第三人意外責任險的保障還是不夠，可以再加保這種險 (通常是第三人責任險的附約)。超額責任險的用途是，當強制加第三人合起來也不夠賠的時候，可以有再多一層的保障。很好的一個例子就是去撞到幾百萬的名車，光賠人家修車費可能就把財損的部份用光了，這個時候就可以用超額責任險這部份的保額來賠，多一層保障總是好事 (況且在台灣百萬名車可是滿街跑)。這種商品還不算太貴，500 萬的保額大概快 2000 元，不過絕對比第三人責任財損部份的保費要便宜 (以相同保額而言)，建議那邊不用買太多 (比如說 30 萬)，用這裡的來補會省一點。

以我自己而言，我買了 300/1500 (人) + 50 (物) 的第三人責任險，再加保一個 500 萬的超額責任險，這樣的保障應該是足夠讓人安心的 (當然不是安心亂開)。

駕駛人傷害險、乘客責任險

第三人責任險只保障對方 (也就是除了坐在我們自己車子內面以外的所有人)，那坐在我們車上的人怎麼辦? 還有，開車的駕駛怎麼辦?

當然，還有強制責任險，強制險的保障包含了所有人 (駕駛及乘客當然也算)，但是如果擔心強制險的保障不夠，可以加買這兩種險來作補強 (不要問我為什麼要弄這麼複雜)。我想，會坐在我們車裡的不是親人就是好友，萬一讓人受了傷本來就已經很不好意思了，出多一點醫藥費、慰問金也是應該的。當然，如果你的車子從不載人，那這部份就可以省下來了。

這保費也不貴，大概 1500 左右就差不多了。

車體險

這個就是大家都很熟悉的那個甲式、乙式、丙式，乙式比丙式貴，甲式又比乙式貴 (而且貴到不像話)，大概就是這樣 … 三者的差別主要在保障的範圍，甲式幾乎什麼都賠 (也就是車子有損傷就賠)，丙式只有在車撞車的情況才賠 (換句話撞到電線杆或是撞到牛不賠)，乙式則介於兩者中間 (比丙式多了火災、雷擊、墜落物等)。

車體險保費高低[⁹]跟車子 (廠牌、型號、年份) 還有車主性別、年齡及駕駛紀錄都有關係，跟強制險一樣，如果沒有出險紀錄也可以有折扣，這邊大概講一下它是怎麼計算的。車子還有車主的部份是沒有折扣的，一個表格定好，是什麼車就是什麼係數，車主幾歲、是男還是女就是固定的係數，這沒什麼好說的。有折扣的是無肇事減費的部份。首先它會看三年內有無出險紀錄，如果有一次就不打折，有兩次保費要加 20%，有三次加 40%，有四次加 60%，以此類推 (無上限)。另一方面它還會再依照連續無出險年數來打折，一年無出險紀錄扣 20%，連續兩年則扣 40%，三年扣 60% (最高只算到三年)。把這兩部份加起來就是最後的無肇事減費係數。

舉個例子好了，假設我前年買的新車，去年底被保險員騙了出險一次作美容 (該死)，今年學乖了不出險，那明年的無肇事減費係數就是 0% (出險過一次) — 20% (一年內無出險) = -20%，所以是打八折。但是如果運氣不好 (一樣去年被騙出了一次險作美容)，今年因為意外又出了一次險，係數會變成 20% (出險過二次) — 0% (沒有連續年度無肇事紀錄) = 20%，所以是要多付 20%! 相同的情況，但是如果去年沒有被騙出險作美容，就算今年運氣不好要出險，現在的係數也只會是 0% (出過一次險) — 0% (沒有連續年度無肇事紀錄) = 0% 而已，有差哦，如果保的是甲式的，那可能會差上好幾萬!

看起來實在有點複雜，其實說穿了真的很簡單，反正沒事別亂出險就是了 (因為只要有出險就一定會貴到隔年的保費)。不要再相信業代沒有根據的話了，羊毛出在羊身上!

竊盜險

竊盜險保費的算法，跟車子本身的價值有關，而且車子的價值是要算折舊的，比如說新車 100 萬買來，大概三年其價值就會折到 50% 左右 (每個廠牌、車款折舊速度都不大一樣)。通常大家的講法是新車幾年內才比較需要保竊盜險，等車子老了也沒多少價值了，保了也覺得浪費。

這個應該不用多講了，怕車被牽走就保吧!

車險的折扣

前面已經講過，只要保持沒有肇事的紀錄，強制責任險以及車體險都可以有折扣 (其實第三人意外責任險也有[10])，而且這個折扣是公定的，好比說強制責任險費率是法律規定的，保險公司沒幫你折掉可以去申訴。除此之外，如果固定都跟同一家保險公司或是業代保，可能還有額外的折扣 (也許是業代的傭金部份多少給一點優惠)，像我們的就有再打九折。

駕駛是誰有差

這部份只有車體險需要考慮，其他不管是強制責任險、第三人意外責任險、竊盜險，不論出事的時候開車的是誰，都有理賠。但是車體險有點不一樣哦，除非開車的是被保險人 (車主或是有特別列名的人)，或是車主的配偶、同居家屬、四等親以內血親或是三等親以內姻親，否則車體險是不會理賠的。最常看到的情形就是男女朋友，如果是另一半開車去撞到，保險公司還是會跟他(她)索賠的，所以在這種情況下一定要記得把另一半 (男女朋友) 特別列為被保險人。

所以啦，車子不要隨便借人開，不要以為車子有保險了就萬事 OK …

其實現在還有另一種特別的 “限定駕駛人” 車體險，保費比較便宜 (大概是正常的 65%)，它規定駕駛人必須是車主或是其配偶，否則一律不予理賠。車體險很貴，如果你的愛車不會借給別人、都是自己夫婦倆在開，這部份的錢倒是可以省下來 。

黃金組合

好吧，開車很累，大家都比牛仔還累，有時間還是多幫自己及家人想想，車險是不是有達到自己心目中的黃金組合:

• 強制責任險記得要買，免得被罰錢。

• 第三人意外責任險一定要加，體傷的部份至少加到 200 萬，不過最好可以到 300~400 萬以上，財損的部份至少 30 萬。記得選擇多倍型的，至少要五倍。加個超額責任險 300~500 萬。加個乘客責任險，額度至少 10 萬吧。如果自己本來沒保意外險 (唉呀，怎麼會沒保)，就再加個駕駛傷害險。

• 車體險要看人吧，如果覺得自己開車都很小心，也很謹慎，從來也沒出過什麼大事，也許不保也可以，因為就算出了事了不起就自己付錢修車，沒啥大不了 (這跟要賠人家的部份是不一樣的)。但要是你視車如愛妻，開車又常常出事 (這有點矛盾?)，開的是百萬名車，保個甲式的可能會比較安心一點。

• 竊盜險，如果是三年內新車就買吧，幾千塊對賭幾十萬 (國產車) 也還划算 (一切都是萬一啊)。如果有自己的車庫，又很少停在外面，或是車子開好幾年了早就折舊到剩沒幾萬塊，這部份也許就不太需要。

[¹]: 未投保汽（機）車強制責任保險中，會遭受如何之處罰?

[²]: 強制險提供如何的保障?

[³]: 一車輛事故

[⁴]: 那些情形世紀產物保險公司賠償後，會對肇事加害人進行追償?

[⁵]: 強制汽車責任保險費率表, 95年強制汽車責任保險費率表

[⁶]: 車禍慰問金，紅包險，理賠最多5萬元

[⁷]: 100/200 是指最高每一位受害人可以獲得 100 萬的賠償，每一件事故最高上限 200 萬 (不管幾人)。

[⁸]: 你也許會覺得奇怪，為什麼不直接把雙倍型的直接買到 750/1500 就好? 因為那樣的保費會比買五倍型的 300/1500 要貴上很多 (至少不是差個幾百塊而已，一定是好幾倍以上)。

[⁹]: 車體險保費算法

[¹⁰]: 車體險、強制險、第三人意外責任險保費算法

是的，我沒有保任何醫療險，不過為了不想一再解釋我的理由，我決定來好好試算一下，這樣子以後再遇到有人問起，直接叫他來看這篇就行了!

要回答這個問題，其實還真不容易 (要不然保險公司就不會這麼賺錢了)，所以這篇文章會有點長，但是沒有耐心看完的人可以先偷看一下我的結論:

* 絕對不要買終身醫療險，不管是無上限還是有上限的。
* 不過，最好買個定期醫療險。

這就是我的最終結論，不過底下還有很多但書及分析，建議大家還是好好看下去，免得作了決定以後還跑來怪我

那麼，終身醫療險到底有什麼不好? (既然我說不要買)

所謂的終身，指的就是她的保障是一輩子的，一般而言，終身醫療險還可以分為兩種，一種是沒有給付上限的 (無上限)，也就是說只要你符合理賠的標準，保險公司就得一直付給你 (直到你有幸”回老家”)。另一種則是有給付上限的 (也就是所謂的”帳戶型”)，不管你申請理賠幾次、每一次請了多少錢，反正保險公司最多最多全部加起來總共就只會付你付到一定的額度，好比說是一百萬 (當然這個數目得看你到底是保多少)。其實這樣講起來，帳戶型也不見得就可以被稱作是”終身”醫療險了，因為很有可能我們人還沒老就先生了場病，一下子把所有的額度都給用完了。

我想，只要是正常人就一定會選無上限的終身醫療險 … 理由很簡單嘛，有給付上限，萬一我老了真的需要用的時候額度不夠 (好比說六十歲前不小心生個病就用掉了四十萬，結果額度只剩六十萬，那六十歲以後不就得省著點用了)，那我保這個險幹嘛?!

不過這裡有個 -好- 壞消息: 過了今年以後，無上限終身醫療險就會變成歷史名辭了[¹]，也就是說，保險公司以後不會再賣了。既然買不到了，我想也不用特別去分析，這裡我就只拿帳戶型的來分析 (我比較懶，反正兩個都不划算，挑簡單一點的來比較就好了)。

底下我拿南山人壽的終身醫療險來作例子，用男性、30 歲、20 年期、10 單位 (日額 1000) 來做試算:

終身醫療保險 v.s. 自行準備醫療金

這個險啊，簡單來講就是你每年付給南山 $27880，連續付個 20 年，全部付完之後南山就保你一輩子 (但是最高只能賠到 100 萬)。換算一下，等於我們總共付出去 $557600，最後換回來 100 萬的保障，看起來還不賴嘛，不是嗎? 大概有 80% 左右的獲利耶~

但是真的是這樣嗎?

保險公司算給你看的結果，通常會選擇性的遺漏了幾個很重要的地方，第一個就是忽略了通膨對保額的影響。什麼是通膨[²]? 只要回想一下小時候一碗陽春麵只要 15 元就知道了 (好啦，你吃到的是 20 元，OK?!)，現在一碗要多少? 正常情況下，我們手上的錢會愈來愈薄，也就是說，一樣是新台幣 15 元，20 年前至少還吃得起一碗麵，現在可能連一盤小菜都叫不起! 回過頭來從保額的角度來看，現在的 100 萬看起來還不少，但是 20 年後可能就不算什麼大錢了 …

所以啦，上面那個表第六欄，我把通膨的因素也考慮進來，看看對 100 萬的保額有什麼影響。我只用平均一年 2% 的通膨來計算，以過去二、三十年的統計來說，這已經算是低標了 … 我們來看看繳完 20 年後，因為通膨的關係，本來的 100 萬其實只剩 67 萬的價值了 (也就是說，一百萬在 20 年後用起來，其實跟今天用 67 萬的感覺是一樣的)。這其實不難理解，像這一、兩年所有物價都漲個不停 (油價、小吃店、衛生紙、門診費 …)，本來兩年前看個病只要兩、三百塊就可以解決，現在至少是三、四百元起跳，天知道二、三十年以後看個病要花多少錢啊!

所以啦，30 歲的時候想說買個終身醫療險，以為可以買到 100 萬的保障，其實到了 50 歲只剩下相當於 67 萬的保障 … 更慘的還在下面，因為是終身的關係，還得繼續通膨下去，60 歲時只剩 55 萬，70 歲時只剩 45 萬，80 歲時只剩 36 萬 … 好慘! 不要忘記哦，這可是在年輕時 (30 至 50 歲之間) 用了總共 $557600 買來的保險啊~

差別在哪裡? 差別在這 100 萬的保額沒有利息，不像存在銀行或是自行理財，這 100 萬是死的，會隨著時間的流逝、通膨的侵蝕，慢慢的就沒了

讓我們換個角度來想，如果我們 30 歲的時候不買這個終身醫療險，改用自己理財的方式來籌備自己的醫療準備金，結果會有什麼不一樣? 先保守一點好了，找個定存 (這夠保守了吧!)，用功一點去找個年利率 2.55% 的定存，每年存相同的 $27880 進去，連續 20 年。請注意，在這裡我也把通膨給算進來，所以 2.55% 的定存實際上只剩 0.55%，這樣比較起來才公平。我們來看一下定存的威力有多大: 50 歲的時候可以累積到 59 萬，嗯，還不錯，雖然還是輸給終身醫療。不過 60 歲呢? 62 萬哦，就比終身醫療險通膨到這個時候的 55 萬要高了不少呢! 就算是最最最保守的定存，到了大概 55 歲時也都可以輕輕鬆鬆贏過終身醫療險! 再繼續往下看呢? 真的是活的愈久，保終身醫療險就愈不划算啊!

不過呢，沒有人這麼保守的啦，定存是上個世紀在用的理財方式，用功一點，很輕鬆就能找到投資酬率比定存更高的好標的。比如說買積優股每年來領股息，這樣子的股票很多啊，最大的要求就在穩定性，台塑集團、中鋼、或是台灣 50 等都很適合，每年殖利率少說也有 4% 的水準，我們就拿 4% 來試算好了 (一樣，我們扣掉通膨的 2%): 50 歲的時候就贏過終身醫療險了耶~ 到了 60 歲的時候可以多出快 30 萬，70 歲的時候可以多出快 60 萬，80 歲的時候多出快 90 萬!

試算表後面還有 6% 跟 10%，給各位參考一下，講老實話，6% ~ 10% 也還不算非常貪心的投資目標，所以我說終身醫療險真的是很不划算，也難怪保險永遠是最賺錢的行業之一!

這樣有沒有一點心得了? 有錢應該要想辦法讓錢自己去生錢，我沒有叫大家都去炒股票、玩期貨，不過即使是放定存也要比買終身險好 (還是盡量不要，太浪費了，起碼買 REITS 或是台灣 50)。特別是我們這裡談的是自己將來要使用的醫療準備金，數字差這麼多，將來可以獲得的醫療品質相對的也就會差這麼多!

還不相信?! 再說幾個終身醫療險不好的地方加強一下各位的信心。除了給付總額有上限外，她還有日額的限制，像我們這裡試算就是用日額 1000 來算的，也就是說保險公司最多一天只付 1000 元 … 當然還有實支實付跟日額型的分別，不過這裡就簡化一點好了。要是我哪天生病住院，花了 1500 元，抱歉，那多出來的 500 元得自己出，請不到理賠的。當然，日額是可以隨自己喜歡調整的，我們也可以選擇要日額 2000，只不過要繳兩倍的錢就是了 … 除此之外，醫療險還有很多其他有的沒的制限，像每一次手術可以給付的上限，還有到底什麼病有理賠、什麼病又沒有理賠等等。我要講的是，雖然有 100 萬的保障在保險公司那裡，但是一想到每次想動用都得申請，而且還有一大堆的條款限制，我就覺得還不如把錢放在自己身邊，需要用的時候自由動用，不需要的時候又能經由理財的方式增值 (而不是放在保險公司那只有慢慢貶值的份)，這樣不是比較好嗎!?

好吧，到這裡完成了我的第一個結論 …

> 與其買終身醫療險，不如把那個錢留下來，就算是放定存也好，很輕鬆的就能夠獲得比買終身醫療險更高的保障 (也就是可以存下更多的錢啦!)

再來，定期醫療險有什麼不一樣? 為什麼我會說最好要買?

照著我的第一個結論做，最大的風險就是”萬一”運氣實在不好，三十幾、四十歲就得了重病，生活上可能就會發生困難，有可能因為生病而丟掉工作、沒有收入，也可能因為要治癒而得花掉大部份的積蓄。問題在這個階段醫療準備金才剛開始起步根本還不夠充裕 (用 4% 投資報酬率來算，40 歲時只累積到 31 萬)，就算剛好夠用、可以把病醫好，但是一切又得重來 (醫療金歸零了得重新累積)，這樣子以後的生活會變得很慘!

不過這就是生活的風險，其實也就是為什麼要保”險”的真正目的。換句話說，醫療險的確有他的必要性，但是也不是說就一定要買終身型的醫療險，因為其實還有另一種類型的：定期醫療險。

所謂的定期，是指這種保險的保障只有一定的期間 (通常都是一年一期)，時間到了就得再買一次。不過也就因為保險期間固定 (對保險公司而言風險比較小)，保費相對而言也就便宜多了，通常一年的定期醫療險費用只要數千元，跟終身醫療險動輒數萬比較起來差很多，但是在保障上則是沒什麼差別 (當然要看個別商品的內容而定)。

所以啦，針對我們剛剛提到的問題，只要配合定期醫療險就能獲得保障了。讓我再來試算一下，把買終身醫療險的錢省下來自己理財，再加買定期醫療險會是個什麼樣的情況:

終身醫療險 v.s. 定期醫療險 + 自行準備醫療金

簡單來說，我從每一期的 $27880 裡拿出 $2100 來投保定期醫療險，剩下的 $25780 存起來作理財投資 (定存、買積優股 …)，連續 20 年，不過要注意的是定期醫療險的保費會隨著年齡而略微增加，不過還是會在數千元的範圍內。如果以投資報酬率 4% 來看 (不要再想定存了，那實在是保守過頭了），到了 50 歲的時候，其實自己可以累積到的醫療準備金 61 萬已經跟終身醫療險經過通膨後的 67 萬相差不多了，事實上往下看個幾格，只要再等三年，到了 53 歲的時候就已經贏了哦! 現在我們不但有相同的保障 (透過投保定期醫療險)，而且手上的醫療準備金也高於終身醫療險，雙贏耶! 這樣子還有什麼理由非買終身醫療險不可?

所以啦，我的第二個結論也完成了

> 為了避免運氣不佳的”萬一”發生，最好買個定期醫療險，代價不高卻可以給我們非常大的保障。

定期險 (不管是醫療險還是壽險) 付出去的保費到期之後就拿不回來了，這應該很明顯也很自然。但是很多人對於這點卻無法釋懷，最後往往會轉而投向 (在保險公司的誘導之下 …) 終身險的懷抱。希望這裡的分析可以讓大家對終身險的本質有比較深入的了解，買保險嘛，本來就是買一個保障，萬一不幸有需要用到，那是還好有保! 要是”有幸”平平安安不需要動用，那是老天保佑，更應該高興這保費花得有價值。千萬不要因為這種”心有不甘”的想法而去買一堆根本不是保險的保險!

[¹]: 聯合理財網: 無上限終身醫療險 熱賣

[²]: Wikipedia: 通貨膨脹