tag:blogger.com,1999:blog-65818020473540100812024-03-28T08:34:58.126+08:00czetsuyatech This blog contains tutorials, how-to's, and tips for programmers, engineers, and anyone else who would like to learn programming and tools/plugins used by professional programmers.avcarreonhttp://www.blogger.com/profile/07645787568344634224noreply@blogger.comBlogger761125tag:blogger.com,1999:blog-6581802047354010081.post-21544468630664901472024-03-24T09:27:00.002+08:002024-03-24T09:27:23.681+08:00How to Transfer Money from Coins PH to Binance<h2 style="text-align: left;">1. Introduction</h2>Worried about the risk of falling victim to scams while engaging in peer-to-peer (P2P) transactions for buying and selling coins? While there are several strategies to mitigate this risk, it's important to note that transfer fees can often be costly. One platform I've found to have reasonable transfer fees is ARBITRUM. For the purpose of this exercise, let's consider transferring a USDT coin from Coins PH to Binance.<div><br /></div><div>For this exercise, you need a Binance and Coins PH accounts.</div><h2 style="text-align: left;">2. Preferring for Transfer - Binance</h2><div>We need to get the USDT wallet address in preparation for the transfer.</div><div><br /></div><div>2.1 From the menu, select Spot.</div><div><br /></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDhlY7f9e_SIDbpq2Zi_2r3fCVpGnUPPFOZsYnxrBqvH9QJf9IJ7U0N1uHjEnPBRMpuMJkrzPNxFOxsosiH33fSL4w0y0ZvK9d8GK5qAfBf3smhEmhW_iLJ0Up6Q4BnObJbxqp93vopYb_foyamcQrBsoNZufX_B9H7cHSP5kpL5PGvgSjC3FFrL3y20Nd/s821/1-binance-spot.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="547" data-original-width="821" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDhlY7f9e_SIDbpq2Zi_2r3fCVpGnUPPFOZsYnxrBqvH9QJf9IJ7U0N1uHjEnPBRMpuMJkrzPNxFOxsosiH33fSL4w0y0ZvK9d8GK5qAfBf3smhEmhW_iLJ0Up6Q4BnObJbxqp93vopYb_foyamcQrBsoNZufX_B9H7cHSP5kpL5PGvgSjC3FFrL3y20Nd/s16000/1-binance-spot.png" /></a></div><div><br /></div><div>2.2 Find the USDT coin, click the "..." menu, and select "Deposit".</div><div><br /></div><div><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXByWSreNHey4FqFazCFdpKElXLcEsbsGv6IiPOLD30dPLlF1QP_sK-HCteJu_UsJ24yam6z-NoaOtKDtEepPGctozF5PtcnMna9nd3eViu3HsOCTMv0Da4pNXo0yL-VyfdC7YmumVuhyphenhyphen4q877JhslvaPO4kQewremGBxLTTMcx0Di6YdHYZBjeTg6BrgC/s2052/2-USDT-wallet.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="692" data-original-width="2052" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXByWSreNHey4FqFazCFdpKElXLcEsbsGv6IiPOLD30dPLlF1QP_sK-HCteJu_UsJ24yam6z-NoaOtKDtEepPGctozF5PtcnMna9nd3eViu3HsOCTMv0Da4pNXo0yL-VyfdC7YmumVuhyphenhyphen4q877JhslvaPO4kQewremGBxLTTMcx0Di6YdHYZBjeTg6BrgC/s16000/2-USDT-wallet.png" /></a></div><div><br /></div><div>2.3 Select the ARBITRUM network, and copy the the deposit address.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiYP4tUaFA2T2vnExg5tTvMMqFZljt_JRN6p16XPVUtEAKMW39BJUFMw4stxzInOjf_e1JvIpX-dm7mHqq87rSge95JlPrquf8523OVi7FXHY9JFWAHrsjiEQK0D_9NIPkH3PN1RF8-slDqqWdCuDrhO4f_C4CI4EPaYpsAHfNqelQfbFgh2dX6p3QhJPI/s1263/3-Binance-export-wallet-address.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1263" data-original-width="1177" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiYP4tUaFA2T2vnExg5tTvMMqFZljt_JRN6p16XPVUtEAKMW39BJUFMw4stxzInOjf_e1JvIpX-dm7mHqq87rSge95JlPrquf8523OVi7FXHY9JFWAHrsjiEQK0D_9NIPkH3PN1RF8-slDqqWdCuDrhO4f_C4CI4EPaYpsAHfNqelQfbFgh2dX6p3QhJPI/s16000/3-Binance-export-wallet-address.png" /></a></div><br /><h2 style="text-align: left;">3. Making the Transfer in Coins PH</h2><div>3.1 Go to your account portfolio. Find USDT and select Send USDT.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9dxoWwgxCbWGuHn2sM-myZyooC5Ejs5WFqkuv6rDyrqaCVgIncEmjW1nr-blj1oaejWgHxQ1H2b5fCC3GtMKHoicJPDB35OGs-5FK0VNFknuNINvnerftMZLnO4V1Sj4r4IN19at-2m9u9C573QvPAEbqxUGFZJbAuSba5aINzm-ahGGkez3NzMPXKwTP/s1617/4-CoinsPH-portfolio.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="974" data-original-width="1617" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9dxoWwgxCbWGuHn2sM-myZyooC5Ejs5WFqkuv6rDyrqaCVgIncEmjW1nr-blj1oaejWgHxQ1H2b5fCC3GtMKHoicJPDB35OGs-5FK0VNFknuNINvnerftMZLnO4V1Sj4r4IN19at-2m9u9C573QvPAEbqxUGFZJbAuSba5aINzm-ahGGkez3NzMPXKwTP/s16000/4-CoinsPH-portfolio.png" /></a></div><div>3.2 Select the Arbitrum network to match what we selected from Binance.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMdvSeDY7VzqCiQMLYBWtddbpW7QX0DFmIpZiqOU5Mc2su9jRzct1CWj8_105zfrxJkohBlIBI09snAs562vrmDnFtDYyjDlmm3qU3iK1YQIJoouk9QzHW5pW76cbiuay1kAwqzZT6Q_4-wJ5L3rf9vYOeDtQHumPviV7yDBbxMPLdXpL89FavJbdF90Ek/s1058/5-CoinsPH-arbitrum-network.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1058" data-original-width="1042" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMdvSeDY7VzqCiQMLYBWtddbpW7QX0DFmIpZiqOU5Mc2su9jRzct1CWj8_105zfrxJkohBlIBI09snAs562vrmDnFtDYyjDlmm3qU3iK1YQIJoouk9QzHW5pW76cbiuay1kAwqzZT6Q_4-wJ5L3rf9vYOeDtQHumPviV7yDBbxMPLdXpL89FavJbdF90Ek/s16000/5-CoinsPH-arbitrum-network.png" /></a></div><div><br /></div><div>3.3 Specify the amount that you want to send.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6O0LA7ZcFLLwGqa3nBEgP6bZUoq6zlgLGcp5m3apmXSAZ7jOcm9aewHRq_BDoIWKkTlsXfd9kF30-fwJ1M1QMd2v6rqyeZ5BKwn-wmB77EeqZshoxXi5c_-DWOn1drssbwI7PuclCZFUMlQIyV8g_zDzwEehkLJeRhk2L-eknV31EhLtxmcquNakov2ur/s1858/6-CoinsPH-send.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1858" data-original-width="1048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6O0LA7ZcFLLwGqa3nBEgP6bZUoq6zlgLGcp5m3apmXSAZ7jOcm9aewHRq_BDoIWKkTlsXfd9kF30-fwJ1M1QMd2v6rqyeZ5BKwn-wmB77EeqZshoxXi5c_-DWOn1drssbwI7PuclCZFUMlQIyV8g_zDzwEehkLJeRhk2L-eknV31EhLtxmcquNakov2ur/s16000/6-CoinsPH-send.png" /></a></div><br /><div>3.4 You will be presented with a list of checklists, once verified press "I Understand".</div><div><br /></div><div>3.5 In the next screen, you will be asked to confirm your transaction details.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbEV0xhtUuM5mgPTYK1EqgjVUEcib6c5N76vO5MQF9UG8a5oC6QlzAOYRvNON8SZAZDEOLtJchJU5Jqtc2GeYOnpiThgLWDDs8ThujNrkVyyF7H7VvR3BNbBHFv5jO20TUQqDO0__eVmE_LYVKepWJReIFOCOreWwMdEqMvancmP5aMZychqGq_1aC7xYd/s1462/7-CoinsPH-confirm-tx.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1462" data-original-width="1028" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbEV0xhtUuM5mgPTYK1EqgjVUEcib6c5N76vO5MQF9UG8a5oC6QlzAOYRvNON8SZAZDEOLtJchJU5Jqtc2GeYOnpiThgLWDDs8ThujNrkVyyF7H7VvR3BNbBHFv5jO20TUQqDO0__eVmE_LYVKepWJReIFOCOreWwMdEqMvancmP5aMZychqGq_1aC7xYd/s16000/7-CoinsPH-confirm-tx.png" /></a></div><br /><div>3.6 The next step would be for you to validate the transaction by entering the security code sent via email.</div><div><br /></div><div>3.7 The transaction will be processed and after a minute or two you should receive a successful message.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHAF7EG8t2pSrc6fikU8bCS4ncx47-WP-hg9k7Th_C6xKivtVAOok5RnO2MB3u59dlzWUWIClj0btLFAdtRQD-t23Hp4P4Oy-EgY0DrcaaL7ZI01JFdKV41DV0Yi-RX0EmVuBPDkOWbE58KNGgzLpKSHGAOrvgo4PUc_HrSJGc1dL_xn6tzw6OK5GPcZmf/s2062/8-CoinsPH-transfer-status.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1250" data-original-width="2062" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHAF7EG8t2pSrc6fikU8bCS4ncx47-WP-hg9k7Th_C6xKivtVAOok5RnO2MB3u59dlzWUWIClj0btLFAdtRQD-t23Hp4P4Oy-EgY0DrcaaL7ZI01JFdKV41DV0Yi-RX0EmVuBPDkOWbE58KNGgzLpKSHGAOrvgo4PUc_HrSJGc1dL_xn6tzw6OK5GPcZmf/s16000/8-CoinsPH-transfer-status.png" /></a></div><div><br /></div><h2 style="text-align: left;">4. Disclaimer</h2><div>The transaction fee is subject to change without prior notice. Therefore, it's important to verify the current rate beforehand and conduct a test transfer before proceeding with a large amount.</div>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-33642318994773705932024-03-07T08:44:00.005+08:002024-03-07T12:44:04.314+08:00Hands-On Coding: Exploring Hyperparameters for Programmers<h2 style="text-align: left;">Introduction</h2>
<div>In this article, we will explore different techniques for finding the optimal hyperparameter values from a given set of parameters in a grid. Particularly we will look at RandomizedSearchCV, GridSearchCV, and BayesSearchCV.</div>
<div>
<br />
</div>
<div>In this blog you will learn:</div>
<div>
<ol style="text-align: left;">
<li>How to initialize the parameter grid.</li>
<li>How to find the optimal hyperparameters based on a given technique.</li>
<li>How to build a model (XGBClassifier) to use the hyperparameters.</li>
<li>How to score the performance of the model.</li>
</ol>
</div>
<h2 style="text-align: left;"> RandomizedSearchCV </h2>
<pre class="brush: java" style="text-align: left;">param_grid = {
"gamma": [0, 0.1, 0.2, 0.5, 1, 1.5, 2, 3, 6, 12, 20],
"learning_rate": [0.01, 0.02, 0.03, 0.05, 0.1, 0.2, 0.3, 0.5, 0.7, 0.8],
"max_depth": [1, 2, 3, 4, 5, 6, 8, 12],
"n_estimators": [25, 50, 65, 80, 100, 115, 200]
}
grid_search = RandomizedSearchCV(estimator=classifier_0, param_distributions=param_grid, scoring=scoring)
</pre>
<h2 style="text-align: left;"> GridSearchCV </h2>
<pre class="brush: java" style="text-align: left;">param_grid = {
"gamma": [0, 0.1, 0.2, 0.5, 1, 1.5, 2, 3, 6, 12, 20],
"learning_rate": [0.01, 0.02, 0.03, 0.05, 0.1, 0.2, 0.3, 0.5, 0.7, 0.8],
"max_depth": [2, 3, 4, 5, 6, 8, 12],
"n_estimators": [25, 50, 65, 80, 100, 115, 200]
}
grid_search = GridSearchCV(estimator=classifier_0, param_grid=param_grid, scoring=scoring)
</pre>
<h2 style="text-align: left;"> BayesSearchCV </h2>
<pre class="brush: java" style="text-align: left;">param_bayes = {
'gamma': Categorical(param_grid['gamma']),
'learning_rate': Categorical(param_grid['learning_rate']),
'max_depth': Categorical(param_grid['max_depth']),
'n_estimators': Categorical(param_grid['n_estimators'])
}
grid_search = BayesSearchCV(estimator=classifier_0, search_spaces=param_bayes, scoring=scoring, n_jobs=-1, cv=10)
</pre>
<h2 style="text-align: left;"> Finding the Best HyperParameters </h2>
<pre class="brush: java">best_model = grid_search.fit(X_train, y_train)
hyperparams = best_model.best_params_
</pre>
<h2 style="text-align: left;"> Building and Scoring the Classifier using the HyperParameters </h2>
<pre class="brush: java" style="text-align: left;"># Fitting the Model
ne = hyperparams['n_estimators']
lr = hyperparams['learning_rate']
md = hyperparams['max_depth']
gm = hyperparams['gamma']
print("Recommended Params >>", f"ne: {ne},", f"lr: {lr}", f"md: {md}", f"gm: {gm}")
# Build Classification Model
classifier_1 = XGBClassifier(
base_score=0.5,
colsample_bylevel=1,
colsample_bynode=1,
objective=objective,
booster="gbtree",
eval_metric=eval_metric_list,
n_estimators=ne,
learning_rate=lr,
max_depth=md,
gamma=gm,
subsample=0.8,
colsample_bytree=1,
random_state=1
)
# Fit Model
eval_set = [(X_train, y_train)]
classifier_1.fit(
X_train,
y_train,
eval_set=eval_set,
verbose=False
)
# Get predictions for training data
train_yhat = classifier_1.predict(X_train)
print("Training Preds: \n", train_yhat[:5])
# Set K-Fold Cross Validation Levels
cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=1)
# Training Results
train_results = cross_val_score(classifier_1, X_train, y_train, scoring=scoring, cv=cv, n_jobs=1)
# Brief Review of Training Results
print("Average Accuracy K-Fold: ", round(train_results.mean(), 2))
print("Std Deviation K-Fold: ", round(train_results.std(), 2))
print("Precision Score 0: ", round(precision_score(y_train, train_yhat, average=None)[0], 3))
print("Precision Score 1: ", round(precision_score(y_train, train_yhat, average=None)[1], 3))
</pre>
<h2 style="text-align: left;"> Performance </h2>
<div>Machine: Laptop </div>
<div>Processor: AMD Ryzen 7 </div>
<div>OS: Windows </div>
<div>DataFrame Shape: (7282, 17)</div>
<br />
<table style='box-shadow: 0 0 20px rgba(0, 0, 0, 0.15)'>
<tbody>
<tr style='background-color: #3d98d5; padding: 10px; color: white;'>
<th>Technique</th>
<th>Time (s)</th>
<th>Avg Accuracy K-Fold</th>
<th>Std Deviation K-Fold</th>
<th>Precision Score: 0</th>
<th>Precision Score: 1</th>
</tr>
<tr>
<td>RandomizedSearchCV </td>
<td>17.02</td>
<td>0.54</td>
<td>0.07</td>
<td>0.576</td>
<td>0.601</td>
</tr>
<tr>
<td>BayesSearchCV </td>
<td>105.39</td>
<td>0.52</td>
<td>0.05</td>
<td>0.589</td>
<td>0.568</td>
</tr>
<tr>
<td>GridSearchCV </td>
<td>9413.60</td>
<td>0.53</td>
<td>0.06</td>
<td>0.605</td>
<td>0.623</td>
</tr>
</tbody>
</table>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-48235895566273639912024-03-02T11:29:00.002+08:002024-03-02T11:29:19.053+08:00Understanding How Scope Affects Values in Your Spring REST Controller<p>Below we explore how a scope annotation affects an instance value in a Spring REST controller.</p>
<p>Each controller is annotated with scope.</p>
<pre class="brush: java">
@RestController
@Scope([SCOPE_VALUE])
public class XXXScopeController {}
</pre>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgk_oXEhZyT_QlSTw19VXRUb4GY9GHmLi7-b44tgQTmVNtM03xjfZR34u2BH6r-UXG5rHgrV-LuiUawLP-TZFKwfeo0_QYuOdsfHAoIW7_Kdl5LmGYNTgZQBfBmLK3TFYOU59PfxBZJkbIgrpoBrzigLvcPZiRZ0S9JjjKDUCjLMerIIn0HZDYOjZvBFOGM/s2958/spring-scopes.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="707" data-original-width="2958" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgk_oXEhZyT_QlSTw19VXRUb4GY9GHmLi7-b44tgQTmVNtM03xjfZR34u2BH6r-UXG5rHgrV-LuiUawLP-TZFKwfeo0_QYuOdsfHAoIW7_Kdl5LmGYNTgZQBfBmLK3TFYOU59PfxBZJkbIgrpoBrzigLvcPZiRZ0S9JjjKDUCjLMerIIn0HZDYOjZvBFOGM/s16000/spring-scopes.png" />
</a>
</div>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-32565422099735700692024-02-24T17:05:00.002+08:002024-02-24T17:05:55.383+08:00How to Convert Vertically Stored Asset Data into Columnar Format for Cointegration Analysis<h2 style="text-align: left;">Introduction</h2><div>This piece of code fetches asset information from a table stored vertically. </div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1VhFt-PwfCCW0UK4eM6EkhpJt5Y3JcSGX1Hlw94wWDch7-3g2BBbmrtHKI-FGAFYtBRMPI7b5tqoT36rJbXwQCLlKSnUcsbXiMtn8BRHUXFvXCouNqUf7GNofZZPuaHGklGnJ0SOVjTLnjEAIj8rvK2_vLM_oolHm0CTGWvpp4BpFMTjv7iPNpdRD3dYf/s1991/pse_stocksquotes.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="886" data-original-width="1991" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1VhFt-PwfCCW0UK4eM6EkhpJt5Y3JcSGX1Hlw94wWDch7-3g2BBbmrtHKI-FGAFYtBRMPI7b5tqoT36rJbXwQCLlKSnUcsbXiMtn8BRHUXFvXCouNqUf7GNofZZPuaHGklGnJ0SOVjTLnjEAIj8rvK2_vLM_oolHm0CTGWvpp4BpFMTjv7iPNpdRD3dYf/s16000/pse_stocksquotes.png" /></a></div><h2 style="text-align: left;">Dependencies</h2><h3 style="text-align: left;">Install the following package.</h3>
<pre class="brush: java">conda install pandas
conda install numpy as np
conda install mysql-connector-python
conda install sqlalchemy
conda install pymysql
</pre>
<p>Hands-on Coding</p><p>Connect to the database</p>
<pre class="brush: java">def query_df(query):
try:
engine_uri = f"mysql+pymysql://db_user:db_pass_123@localhost:3306/tradewise_pse"
db_conn = create_engine(engine_uri)
df_result = pd.read_sql(query, db_conn)
return df_result
except Exception as e:
print(str(e))
</pre>
<p>
Fetching the Dataset</p>
<pre class="brush: java">if not load_existing:
sql_distinct_tickers = "select ticker from candlestick where event_time='2023-12-29' and ticker not like '^%%'"
df_tickers = query_df(sql_distinct_tickers)
df = pd.DataFrame(index=['event_time'])
### Get the candlesticks
for ticker in df_tickers['ticker']:
sql_ticker_col = "select event_time, close from candlestick where ticker='{0}'"
df_temp = query_df(sql_ticker_col.format(ticker))
df_temp.set_index('event_time', inplace=True)
df_temp.rename(columns={'close': ticker}, inplace=True)
df = df.add(df_temp, fill_value=0)
df.to_csv(file_name)
</pre>
<p>Load the dataset from file</p>
<pre class="brush: java">df = pd.read_csv(file_name, index_col=0)
df.drop(index=df.index[-1],axis=0, inplace=True)
</pre>
<p>Drop NA</p>
<pre class="brush: java">df.dropna(axis=1, inplace=True)
</pre>
<p>Print the dataset</p>
<pre class="brush: java">print(f"Shape: {df.shape}")
print(f"Null values: {df.isnull().values.any()}")
df
</pre>
<p>Save to a file</p>
<pre class="brush: java">df.to_csv(file_name)
</pre><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSfeyHtZICnuPLvmU3c2gffwvghMXBtdEEGyAX59_x-avffROB52Oa_XUJg9eohQb91AFkCL_SNU4YooGzxz2rgYQpGL97PNVRrgjimOT0rSAHufdVyEWA_bXA_wjfmRSB7jUNnfkHOkm0wgYDmBwc3JE0oMF-wqnHdnCgOGZBUA4jsggYtoFKY7EQ4nQb/s2349/columnar_assets.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="510" data-original-width="2349" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSfeyHtZICnuPLvmU3c2gffwvghMXBtdEEGyAX59_x-avffROB52Oa_XUJg9eohQb91AFkCL_SNU4YooGzxz2rgYQpGL97PNVRrgjimOT0rSAHufdVyEWA_bXA_wjfmRSB7jUNnfkHOkm0wgYDmBwc3JE0oMF-wqnHdnCgOGZBUA4jsggYtoFKY7EQ4nQb/s16000/columnar_assets.png" /></a></div><pre class="brush: java">This procedure is in preparation for cointegration testing.</pre>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-21034924196643200352024-02-24T16:38:00.002+08:002024-02-24T16:39:17.394+08:00Learn to Incorporate Rolling Hurst Values into Your DataFrame<h2 style="text-align: left;">The hurst function.</h2><pre class="brush: java" style="text-align: left;">def hurst(ts, min_lag=1, max_lag=7):
lags = range(min_lag, max_lag)
tau = [np.sqrt(np.std(np.subtract(ts[lag:], ts[:-lag]))) for lag in lags]
poly = np.polyfit(np.log(lags), np.log(tau), 1)
return poly[0]*2.0
</pre><h2 style="text-align: left;">
Adding to our DataFrame</h2><div>The hurst value is computed with the last 14 close values.</div><pre class="brush: java">df['Hurst'] = df['close'].rolling(14).apply(hurst, raw=True)
df[10:20]
</pre><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgoXLAvUM3MLIXGbFp8daR2v4FJcRbWDPHqYHtEc8YsFM_TDF4Cskty-kgG6LzmU3BgNkBTZOfoIGsGR4DQ3-1UeD4Y8oD39oipEd3e2ioH7HJRNPaP4GlO6wwJU0AgabUJfUy9W4Lqyf91n2xO0L5QLreudVx5o2jwfbTrGEu1SYRiI2FRzMIeFMxXkR-B/s1577/adding_rolling_hurst_values.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1027" data-original-width="1577" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgoXLAvUM3MLIXGbFp8daR2v4FJcRbWDPHqYHtEc8YsFM_TDF4Cskty-kgG6LzmU3BgNkBTZOfoIGsGR4DQ3-1UeD4Y8oD39oipEd3e2ioH7HJRNPaP4GlO6wwJU0AgabUJfUy9W4Lqyf91n2xO0L5QLreudVx5o2jwfbTrGEu1SYRiI2FRzMIeFMxXkR-B/s16000/adding_rolling_hurst_values.png" /></a></div><br />czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-68614017656916589642024-02-01T15:48:00.006+08:002024-02-02T08:53:07.321+08:00Implementing Glowroot: A Hands-On Tech Review for Application Monitoring Mastery<h2 style="text-align: left;">1. Introduction</h2>
<p>In the realms of information technology and systems management, Application Performance Management (APM) involves monitoring and overseeing the performance and availability of software applications. APM aims to identify and diagnose intricate application performance issues to uphold a predefined level of service.</p>
<p>
<b>Glowroot</b> is an open-source APM that facilitates a quicker resolution of application performance issues by helping us pinpoint the root causes. It supports applications running from Java 6 onwards.
</p>
<h2 style="text-align: left;">2. Key Features</h2>
<div>
<ul style="text-align: left;">
<li>
<b>Response Time Breakdown Charts:</b> Visualizes the breakdown of response times for better analysis.
</li>
<li>
<b>Response Time Percentile Charts:</b> Offers percentile charts to understand response time distribution.
</li>
<li>
<b>SQL Capture and Aggregation:</b> Captures and aggregates SQL queries for in-depth analysis.
</li>
<li>
<b>Service Call Capture and Aggregation:</b> Gathers and aggregates data on service calls for comprehensive insights.
</li>
<li>
<b>MBean Attribute Capture and Charts:</b> Monitors and charts MBean attributes for performance evaluation.
</li>
<li>
<b>Configurable Alerting:</b> Allows users to configure alerts based on specific criteria.
</li>
<li>
<b>Historical Rollup:</b> Provides historical data rollup at different intervals (1m, 5m, 30m, 4h) with configurable retention settings.
</li>
<li>
<b>Full Support for Async Requests:</b> Supports asynchronous requests that span multiple threads.
</li>
<li>
<b>Responsive UI with Mobile Support:</b> User-friendly and responsive interface with mobile support for accessibility.
</li>
<li>
<b>Optional Central Collector:</b> Offers the flexibility of an optional central collector for centralized data management.
</li>
<li>
<b>Supports Multiple Application Servers:</b> Wildfly, JBoss EAP, Tomcat, TomEE, Jetty, Glassfish, Payara, WebLogic, WebSphere
</li>
</ul>
<h2 style="text-align: left;">3. Central Collector</h2>
<div>The central collector collects runtime information from the registered services and offers a GUI for easy viewing. <a href="https://github.com/glowroot/glowroot/wiki/Central-Collector-Installation">https://github.com/glowroot/glowroot/wiki/Central-Collector-Installation</a>
</div>
<h3 style="text-align: left;">3.1 Installation</h3>
<div>You can follow the steps above for running the GlowRoot central collector either as a standalone or as a docker image. For this section, I'll share how it can be run locally.</div>
<div>Here's my docker-compose file for running glowroot-central with cassandra. Override username, password, and contactPoints in glowroot-central.properties.</div>
<pre class="brush: java">
version: '3.8'
networks:
tradewise-network:
services:
cassandra:
image: cassandra:latest
container_name: cassandra
restart: unless-stopped
ports:
- "9042:9042"
networks:
- tradewise-network
glowroot-central:
image: glowroot/glowroot-central:0.14.1
container_name: glowroot-central
restart: unless-stopped
volumes:
- ./glowroot-central.properties:/usr/share/glowroot-central/glowroot-central.properties
depends_on:
- cassandra
ports:
- "4000:4000"
- "8181:8181"
networks:
- tradewise-network
</pre>
<h2 style="text-align: left;">4. Instrumentation</h2>
</div>
<div>There are 4 ways in which we can integrate GlowRoot into our services <a href="https://glowroot.org/instrumentation.html">https://glowroot.org/instrumentation.html</a>. For this exercise, we will focus on using the agent API. </div>
<h3 style="text-align: left;">4.1 Editing the pom.xml File</h3>
<div>In our Spring Boot project's pom.xml file, add the GlowRoot agent configuration.</div>
<pre class="brush: java">
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<executions>
<execution>
<id>glowroot-plugins</id>
<phase>validate</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>target/glowroot-plugins/</outputDirectory>
<resources>
<resource>
<directory>glowroot-plugins</directory>
<filtering>false</filtering>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>${maven-dependency-plugin.version}</version>
<executions>
<execution>
<id>copy-glowroot-jar</id>
<phase>prepare-package</phase>
<goals>
<goal>copy</goal>
</goals>
</execution>
</executions>
<configuration>
<artifactItems>
<artifactItem>
<groupId>org.glowroot</groupId>
<artifactId>glowroot-agent</artifactId>
<version>${glowroot-agent.version}</version>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>${project.build.directory}</outputDirectory>
<destFileName>glowroot.jar</destFileName>
</artifactItem>
</artifactItems>
</configuration>
</plugin></pre>
<h3 style="text-align: left;">4.2 Preparing the Dockerfile</h3><div>We need to add the GlowRoot files in the docker image.</div>
<pre class="brush: java" style="text-align: left;">
# Base Image
FROM eclipse-temurin:17-jdk-alpine
LABEL author=CzetsuyaTech
LABEL maintainer=CzetsuyaTech
# Configuration
WORKDIR /
RUN addgroup --system czetsuyatech && \
adduser --system czetsuyatech --ingroup czetsuyatech && \
mkdir -p /glowroot /glowroot/tmp /glowroot/logs /glowroot/plugins && \
echo '{ "web": { "bindAddress": "0.0.0.0" } }' > /glowroot/admin.json && \
chown czetsuyatech:czetsuyatech -R /glowroot && \
chmod -R 777 /glowroot
USER czetsuyatech
ADD --chown=czetsuyatech:czetsuyatech target/glowroot.jar /glowroot
ADD --chown=czetsuyatech:czetsuyatech target/glowroot-plugins /glowroot/plugins
# Service
ADD --chown=czetsuyatech:czetsuyatech target/*.jar app.jar
# Start
ENV JAVA_JAR "/app.jar"
ENV JAVA_OTHERS "-Xshare:off"
ENV JAVA_OPTS ${JAVA_OPTS}
ENV JAVA_MEM ${JAVA_MEM}
RUN echo "exec java $JAVA_MEM $JAVA_OPTS $JAVA_OTHERS -jar $JAVA_JAR"
ENTRYPOINT exec java $JAVA_MEM $JAVA_OPTS $JAVA_OTHERS -jar $JAVA_JAR
</pre>
<h3 style="text-align: left;">4.3 Running the Spring Boot Service</h3>
<div>To enable instrumentation in our instance, we need to specify the javaagent property. And to send information to the central collector we need to specify the central collector and give our instance an agent id.</div>
<pre class="brush: java">
JAVA_OPTS=-javaagent:glowroot/glowroot.jar -Dglowroot.collector.address=localhost:8181 -server -Dglowroot.agent.id=TradewiseAI
</pre>
<h2 style="text-align: left;">5. GUI</h2>
<h3 style="text-align: left;">5.1 Usage</h3>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaxjUvM36zJOd2-i8oDTvR8icamUjS6PHhoIpsfrXOMSxdipyxAORkTgmSGt033C8Ki9KwwFeGE1NbvR7Ns_SWS3HCpWZBBVscZRDALuqXXncEQRG3bdrQR4UktvwX0MHGKxuFejr3xixWePJCO88003vpAOgOAspkbyyT2Loa44o2_VxJND48gVoyI_yl/s3342/web_transactions.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="1875" data-original-width="3342" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaxjUvM36zJOd2-i8oDTvR8icamUjS6PHhoIpsfrXOMSxdipyxAORkTgmSGt033C8Ki9KwwFeGE1NbvR7Ns_SWS3HCpWZBBVscZRDALuqXXncEQRG3bdrQR4UktvwX0MHGKxuFejr3xixWePJCO88003vpAOgOAspkbyyT2Loa44o2_VxJND48gVoyI_yl/s16000/web_transactions.png" />
</a>
</div>
<div>
<h3>5.2 Transactions</h3>
</div>
<div>
<h4>5.2.1 Web</h4>
</div>
<div>
<div>To assess our REST Endpoints' performance, we access the "Web" section. In the provided example:</div>
<div>
<ol style="text-align: left;">
<li>By choosing "Response Time," we can identify which HttpRequests and JDBC Queries are taking longer in the REST Endpoints.</li>
<li>Opting for "Slow Traces" allows us to pinpoint the specific endpoint that consumes more time.</li>
<li>Selecting "Queries" reveals insights into the queries made, indicating that using the count query is more resource-intensive compared to the select query. This information aids in optimizing and refining the performance of REST Endpoints.</li>
</ol>
</div>
</div>
<div>
<h4>5.2.2 Background</h4>
</div>
<div>
<div>To analyze our background performance, we navigate to the "Background" section. In the given example:</div>
<div>
<ol>
<li>Choosing "Response Time" allows us to identify which Job and Hibernate Queries are consuming more time in the background processes.</li>
<li>Opting for "Slow Traces" reveals that some calls take more time, providing insights into areas that may require attention.</li>
<li>Selecting "Queries" and filtering by the select query, we observe that it is called more frequently and takes longer, particularly when filtered by status. This information helps in understanding and addressing potential bottlenecks in the background processes.</li>
</ol>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi76KdeL-qHs036T9G7jP3Fdv5J1sPjQldEfdejFhXqTu2vG3Z4VqY8N694VYGhQTpAmyzBvPHc0FU0gsuHSzYjF9KMKtezpxOD5MmtCgaX6ePBc3rFWEHs0d0xgc8zwN-vwPyCAyuRybqJ8ZIA6otXyCtti21vzv_nfPATdRwid5vrRZFOf2pQKLmHTsEB/s2994/glowroot-background.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="1723" data-original-width="2994" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi76KdeL-qHs036T9G7jP3Fdv5J1sPjQldEfdejFhXqTu2vG3Z4VqY8N694VYGhQTpAmyzBvPHc0FU0gsuHSzYjF9KMKtezpxOD5MmtCgaX6ePBc3rFWEHs0d0xgc8zwN-vwPyCAyuRybqJ8ZIA6otXyCtti21vzv_nfPATdRwid5vrRZFOf2pQKLmHTsEB/s16000/glowroot-background.png" />
</a>
</div>
</div>
</div>
<div>
<h4>5.2.3 Startup</h4>
</div>
<div>
<div>To assess our startup performance, we can utilize the "Startup" section. In the provided example:</div>
<div>
<ol style="text-align: left;">
<li>By choosing "Response Time," we can identify which startup and filter init processes consume more time.</li>
<li>Opting for "Slow Traces" reveals the duration it takes for the context to initialize fully.</li>
<li>Selecting "Queries" provides insights into the database queries. Some queries will be more resource intensive, with fewer calls but longer duration. This information aids in pinpointing specific areas for optimization within the startup processes.</li>
</ol>
</div>
</div>
<h3 style="text-align: left;">5.3 Errors</h3>
<h4 style="text-align: left;">5.3.1 Web</h4>
<div>
<div>To visualize errors in our REST endpoints, we can navigate to the "Web" section. In the following example:</div>
<div>
<ul style="text-align: left;">
<li>Choosing "Error Messages" reveals errors of type XXXException.</li>
<li>Opting for "Error Traces" provides details on the errors.</li>
<li>Clicking on a specific error allows us to view the detailed trace, aiding in the understanding and resolution of the issue.</li>
</ul>
</div>
</div>
<h3 style="text-align: left;">5.4 JVM</h3>
<div>In our services' JVM, we can monitor and analyze memory status. This allows us to gain insights into the memory usage patterns, allocations, and overall health of the Java Virtual Machine, aiding in the effective management and optimization of our services.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKL8sWMZ55vRtxhQC8m2rKPEqcf-UE8D2xbhDVMiB2JQBylCnly1qt-BTvpLOQHkEdH7IrF088oE1u_Arbwe6rOiUboqKPnmjRpcTLE-aLc4_wv6zhTIT0H1EjYMiYOdVkimBV-zx0Q_2FhT4oG552wsp6BuvIIHCkcXSzjW3wYBIpj_Kq5HzYrtGOSAdS/s3007/glowroot-jvm.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="1853" data-original-width="3007" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKL8sWMZ55vRtxhQC8m2rKPEqcf-UE8D2xbhDVMiB2JQBylCnly1qt-BTvpLOQHkEdH7IrF088oE1u_Arbwe6rOiUboqKPnmjRpcTLE-aLc4_wv6zhTIT0H1EjYMiYOdVkimBV-zx0Q_2FhT4oG552wsp6BuvIIHCkcXSzjW3wYBIpj_Kq5HzYrtGOSAdS/s16000/glowroot-jvm.png" />
</a>
</div>
<h3 style="text-align: left;">5.5 MBeanTree</h3>
<div>The MBeanTree functionality in Glowroot proves invaluable in monitoring instance creation. For instance, to track thread-related metrics and identify potential Thread Leaks, users can navigate to the java.lang section, specifically under Threading. This allows for a detailed examination of thread-related information, aiding in the identification and resolution of potential thread-related issues, such as Thread Leaks.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIM2GtuuJGpHAlqMaCg8p5XmqI_TEbTVDOHOA63Ab1IqssFGroEYEiiNYgr4oTOo2G4-8v9kjWbyjH0ep96tDlOYunIF1kQml6uR3wldnGObLfSyr-1Odt3p2e_R7ALPN0_LlHIxR9IbnrRJB8UWYcHGDzUKWgRwEhZWxY3ljGDKEk8rj-rIXkyCLhNQfD/s2239/glowroot-mbean.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="1746" data-original-width="2239" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIM2GtuuJGpHAlqMaCg8p5XmqI_TEbTVDOHOA63Ab1IqssFGroEYEiiNYgr4oTOo2G4-8v9kjWbyjH0ep96tDlOYunIF1kQml6uR3wldnGObLfSyr-1Odt3p2e_R7ALPN0_LlHIxR9IbnrRJB8UWYcHGDzUKWgRwEhZWxY3ljGDKEk8rj-rIXkyCLhNQfD/s16000/glowroot-mbean.png" />
</a>
</div>
<h2 style="text-align: left;">6. Recommendation</h2>
<p>In a typical product infrastructure, Glowroot serves as our APM tool in each microservice. All microservices are instrumented to gather and transmit data to Glowroot, enhancing our understanding of system behavior. We've chosen Glowroot based on various criteria, including its license-free nature, alignment with the Java ecosystem, and straightforward instrumentation for microservices, ensuring a simple ramp-up and configuration process.</p>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-54074629186276045372023-12-14T08:00:00.003+08:002023-12-14T08:00:18.024+08:00Navigating the Code: A Guide to Environment Variables, Configuration, and Feature Flags<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQsgf1TsulQfEa1eVPYjyjr0V9zEWKvH2CH4B4MvGaQW1Rs7gp-gX1eMQB-pBw1xS-a9yK89tYXbHVsiwyvg-PVHQDYK5cmf1pCYwh5dPzpSCTE8JWp3FPXCvkYdbHlF-SbzQTN9utPp5F-kdfaW1Zzp4sJdARoXpqLEIx31PewNksrF27b7oz0jC8uipz/s1920/nihon-graphy-nCvi-gS5r88-unsplash.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1278" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQsgf1TsulQfEa1eVPYjyjr0V9zEWKvH2CH4B4MvGaQW1Rs7gp-gX1eMQB-pBw1xS-a9yK89tYXbHVsiwyvg-PVHQDYK5cmf1pCYwh5dPzpSCTE8JWp3FPXCvkYdbHlF-SbzQTN9utPp5F-kdfaW1Zzp4sJdARoXpqLEIx31PewNksrF27b7oz0jC8uipz/s16000/nihon-graphy-nCvi-gS5r88-unsplash.jpg" /></a></div><p>Determining the optimal location for storing essential information crucial for an application's functionality necessitates thoughtful planning. This guide aims to assist you in making informed decisions about where to store crucial data. Here, the term "application" encompasses any software created with code and deployed on various platforms and runtimes, irrespective of the underlying software architecture.</p><p>A fundamental principle is to avoid hard-coding any values within your application. Instead, adopt the practice of retrieving all configuration settings from a distinct storage system, separate from the deployment environment. This approach allows for seamless modifications to configuration settings even after the application has been deployed, eliminating the need for redeployment. This separation ensures greater flexibility and facilitates the adjustment of critical parameters without disrupting the operational state of your deployed application.</p><p>When considering the storage location for key/value pairs, it's crucial to evaluate the volatility of each pair—how frequently and by whom it might undergo changes. Reflect on the dynamic nature of the data and the responsibilities associated with its modification. This thoughtful approach will guide you in choosing an appropriate storage solution that aligns with the specific needs and characteristics of each key/value pair. By understanding the frequency and ownership of potential changes, you can make informed decisions that optimize the efficiency and maintainability of your data storage strategy.</p><h2 style="text-align: left;">Environment Variables</h2><div><div>Environment variables, characterized by their infrequent changes, typically occur only once per deployment. Common examples include sensitive information like usernames and passwords for databases, which are often stored securely in a Vault. Additionally, environment variables encompass essential details required for the system's initialization, such as environment specifics and the connection details for the application configuration store, which hosts additional settings. By utilizing environment variables for these foundational aspects, you ensure a secure and stable system setup while centralizing critical configuration information.</div></div><h2 style="text-align: left;">Application Configuration</h2><div>Application configuration, in contrast to environment variables, operates independently of deployments. It encompasses supplementary settings for the system that have the potential to change between deployments at runtime. This flexibility allows for real-time adjustments to the application's behavior and features, accommodating evolving requirements without the need for redeployment. By leveraging application configuration for these dynamic settings, you establish a modular and adaptable structure, enhancing the responsiveness of your system to changing runtime conditions.</div><h2 style="text-align: left;">Runtime User Settings</h2><div>Runtime user settings represent the most volatile category, as users can alter any key/value pair at any given moment. Therefore, the system must be designed to accommodate this high level of dynamism. Options for handling such volatility include implementing robust real-time validation mechanisms and ensuring the integrity and security of user-initiated changes. Additionally, the system should provide an intuitive user interface for managing these settings, enabling seamless customization while maintaining overall system stability and security.</div><h3 style="text-align: left;">Here are three options for handling configuration updates:</h3><div><ol style="text-align: left;"><li><b>Re-read Configuration All the Time:</b> Continuously monitor and re-read the configuration, ensuring that the system remains up-to-date with any changes. This approach provides real-time responsiveness to modifications but may impose a continuous processing overhead.</li><li><b>Re-read Configuration at Set Intervals (e.g., Every 15 Minutes): </b>Implement a periodic schedule to re-read the configuration at predefined intervals, such as every 15 minutes. This approach balances responsiveness with reduced processing overhead, making it suitable for scenarios where near-real-time updates are acceptable.</li><li><b>Receive Push Notifications When New Application Configuration Is Available:</b> Set up a push notification mechanism to alert the system when new application configuration becomes available. This approach minimizes the need for continuous or scheduled re-reading, optimizing efficiency by updating the system only when changes occur. However, it requires a reliable notification infrastructure. Choosing among these options depends on the specific requirements of your system, considering factors such as the criticality of real-time updates, resource constraints, and the overall responsiveness desired for configuration changes.</li></ol></div><p></p><ul style="text-align: left;"></ul><p></p>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-13214078324556832432023-12-11T18:18:00.007+08:002023-12-13T19:16:08.621+08:00Navigating Software and Systems Architecture: A Curated Collection of Resources<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDsV_TKFEGAHMLR4zHjOONgczYgCjy0XcnJCcaJbpa1cIArBfoHzm-ESEL496wVzk77smH1gnRUCNjtoPizgcEdLfoKgA048O9pXpG0e_mGyM_Q0fptd2AOLwzNfRwPaXqyW1nLRi4-e9PCeNNlIHLbKbfNQ-TTKdq48-Oh_rbwwAZgZfeQy5TIfs0GNgR/s1920/books.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1026" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDsV_TKFEGAHMLR4zHjOONgczYgCjy0XcnJCcaJbpa1cIArBfoHzm-ESEL496wVzk77smH1gnRUCNjtoPizgcEdLfoKgA048O9pXpG0e_mGyM_Q0fptd2AOLwzNfRwPaXqyW1nLRi4-e9PCeNNlIHLbKbfNQ-TTKdq48-Oh_rbwwAZgZfeQy5TIfs0GNgR/s16000/books.jpg" /></a></div><p>These books serve as valuable references for building software systems:</p><li>Martin Fowler: Patterns of Enterprise Application Architecture</li><li>Robert C. Martin: Clean Architecture: A Craftsman's Guide to Software Structure and Design</li><li>Martin Kleppmann: Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems</li><li>Len Bass, Paul Clements, Rick Kazman: Software Architecture in Practice</li><li>Gregor Hope: Enterprise Integration Patterns</li><li>Erich Gamma: Design Patterns</li><li>Chris Richardson: Microservice Patterns</li><li>Mark Richards: Fundamentals of Software Architecture</li><li>Mark Masse: REST API Design Rulebook: Designing Consistent RESTful Web Service Interfaces and here</li><li>Matthew Skelton: Team Topologies</li><li>Svyatoslav Kotusev: The Practice of Enterprise Architecture: A Modern Approach to Business and IT Alignment</li><br /><h1 style="text-align: left;">Websites</h1><div><ul style="text-align: left;"><li>https://microservices.io - online version of Chris Richardsons' book</li><li>https://www.enterpriseintegrationpatterns.com/patterns/messaging - online version of Gregor Hopes book</li></ul></div>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-39222809467887958052023-09-07T20:42:00.002+08:002023-09-07T20:42:37.968+08:00Personalized Spring Boot Startups: Custom Banners and Build Information<h2 style="text-align: left;">1. Overview</h2><p>By default, when a Spring Boot application starts, it displays a banner. In this article, we will explore the process of creating a custom banner and utilizing it within Spring Boot applications.</p><h2 style="text-align: left;">2. Creating a Banner</h2><p>Before we begin, it's essential to generate the custom banner, which will appear during the application startup process. You can choose to create the custom banner from scratch or leverage various tools designed for this purpose.</p><p>In this example, we are generating CzetsuyaTech's logo from https://devops.datenkollektiv.de/banner.txt/index.html.</p><p>Create a new file "banner.txt" inside the resources folder and copy and paste this content. This will print a colored banner with application information.</p><div style="background-color: #2b2b2b; color: #a9b7c6;"><pre style="font-family: "Fira Code", monospace; font-size: 12pt;"><div><pre style="font-family: "Fira Code", monospace; font-size: 12pt;">${<span style="color: #cc7832;">AnsiColor.BLUE</span>} _____ ${<span style="color: #cc7832;">AnsiColor.WHITE</span>} _ ${<span style="color: #cc7832;">AnsiColor.BLUE</span>} _______ ${<span style="color: #cc7832;">AnsiColor.WHITE</span>} _<br />${<span style="color: #cc7832;">AnsiColor.BLUE</span>} / ____|${<span style="color: #cc7832;">AnsiColor.WHITE</span>} | | ${<span style="color: #cc7832;">AnsiColor.BLUE</span>}|__ __| ${<span style="color: #cc7832;">AnsiColor.WHITE</span>} | |<br />${<span style="color: #cc7832;">AnsiColor.BLUE</span>}| | ${<span style="color: #cc7832;">AnsiColor.WHITE</span>} _______| |_ ___ _ _ _ _ __ _ ${<span style="color: #cc7832;">AnsiColor.BLUE</span>} | |${<span style="color: #cc7832;">AnsiColor.WHITE</span>} ___ ___| |__<br />${<span style="color: #cc7832;">AnsiColor.BLUE</span>}| | ${<span style="color: #cc7832;">AnsiColor.WHITE</span>}|_ / _ \ __/ __| | | | | | |/ _` | ${<span style="color: #cc7832;">AnsiColor.BLUE</span>} | |${<span style="color: #cc7832;">AnsiColor.WHITE</span>} / _ \/ __| '_ \<br />${<span style="color: #cc7832;">AnsiColor.BLUE</span>}| |____ ${<span style="color: #cc7832;">AnsiColor.WHITE</span>} / / __/ |_\__ \ |_| | |_| | (_| | ${<span style="color: #cc7832;">AnsiColor.BLUE</span>} | |${<span style="color: #cc7832;">AnsiColor.WHITE</span>}| __/ (__| | | |<br />${<span style="color: #cc7832;">AnsiColor.BLUE</span>} \_____/${<span style="color: #cc7832;">AnsiColor.WHITE</span>}/___\___|\__|___/\__,_|\__, |\__,_| ${<span style="color: #cc7832;">AnsiColor.BLUE</span>} |_|${<span style="color: #cc7832;">AnsiColor.WHITE</span>} \___|\___|_| |_|<br />${<span style="color: #cc7832;">AnsiColor.BLUE</span>} ${<span style="color: #cc7832;">AnsiColor.WHITE</span>} __/ |<br />${<span style="color: #cc7832;">AnsiColor.BLUE</span>} ${<span style="color: #cc7832;">AnsiColor.WHITE</span>} |___/<br />${<span style="color: #cc7832;">AnsiColor.BRIGHT_GREEN</span>}#application.title# v#application.version#<br />Build: #build.time#<br />Powered by Spring Boot ${<span style="color: #cc7832;">spring-boot.version</span>}</pre></div></pre></div>
<h2 style="text-align: left;">3. Replacing the Variable Information in the Banner.</h2><div>To replace the build information in the banner, we will use the replacer plugin.</div>
<pre class="brush: java" style="text-align: left;"><plugin>
<groupId>com.google.code.maven-replacer-plugin</groupId>
<artifactId>replacer</artifactId>
<version>1.5.3</version>
<executions>
<execution>
<phase>prepare-package</phase>
<goals>
<goal>replace</goal>
</goals>
</execution>
</executions>
<configuration>
<file>target/classes/banner.txt</file>
<replacements>
<replacement>
<token>#application.title#</token>
<value>${project.artifactId}</value>
</replacement>
<replacement>
<token>#application.version#</token>
<value>${project.version}</value>
</replacement>
<replacement>
<token>#build.time#</token>
<value>${maven.build.timestamp}</value>
</replacement>
</replacements>
</configuration>
</plugin>
</pre><h2 style="text-align: left;">
4. Summary</h2><div>In this short article, we have shown how you can set a customized banner for your Spring application.</div>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-64128307544697591322023-08-25T17:27:00.003+08:002023-08-25T17:33:13.060+08:00Kickstarting Your Spring Cloud Journey with the 2022 Skeleton Project<p>In this article, I will share a composition of a Spring Cloud 2022 skeleton project with Spring Boot 3.</p><h2 style="text-align: left;">1. Introduction</h2><p>Spring Cloud 2022 is a long-awaited major release that builds on top of Spring Framework 6.x and Spring Boot 3.x. It needs at least Java 17 to run, which means it is fully compatible with Jakarta framework.</p><p>GIT repository is available below.</p><h2 style="text-align: left;">2. Spring Cloud Microservices</h2><p>The following services demonstrate a basic microservice architecture.</p><h3 style="text-align: left;">2.1 config-service</h3><p>This module depends on <b>spring-cloud-config-server</b> and connects with a git repo to serve the configuration settings. This is very useful if you want your configurations to be loaded from a central repository.</p><h3 style="text-align: left;">2.2 discovery-server</h3><p>This module depends on <b>spring-cloud-starter-netflix-eureka-server</b> where other services can register and be looked up.</p><h3 style="text-align: left;">2.3 api-gateway</h3><p>This module depends on <b>spring-cloud-starter-gateway</b>. It is the gateway for all incoming requests and makes the necessary load balance forwarding to the actual service implementation.</p><h3 style="text-align: left;">2.4 mail-services</h3><p>This module depends on <b>spring-cloud-starter-config</b> to connect to the config server and <b>spring-cloud-starter-netflix-eureka-client</b> to register to the Eureka naming server.</p><p>It also depends on Webflux for providing a REST endpoint.</p><h2 style="text-align: left;">3. Running the Services</h2><p>Make the folder /config-repo a git repo by going inside the folder and running the command below.</p><p>>git init</p><p>The services must be run in this order</p><ul style="text-align: left;"><li>config-server</li><li></li><li>discovery-server</li><li>mail-services</li><li>api-gateway</li></ul><h2 style="text-align: left;">4. Testing the Services</h2><div><div><b>mail-services</b></div><div><br /></div><div>This service runs on port 8001.</div><div><br /></div><div>The actual URL is <i>http://localhost:8001/mails/profiles/1</i></div><div><br /></div><div><b>api-gateway</b></div><div><br /></div><div>This service runs on port 8001. It simply forwards the request to the mail-services.</div><div><br /></div><div>The actual URL is <i>http://localhost:8000/mails/profiles/1</i></div></div><h2 style="text-align: left;">5. Repository</h2><div><ul style="text-align: left;"><li>Sponsor this blog - <a href="https://github.com/sponsors/czetsuya">https://github.com/sponsors/czetsuya</a></li><li><a href="https://github.com/czetsuya/spring-cloud-2022">https://github.com/czetsuya/spring-cloud-2022</a></li></ul></div><p></p><p><br /></p>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-7337912302871129982023-06-28T16:49:00.005+08:002023-06-28T16:50:53.781+08:00Java XML Validation: How to Validate an XML Document against an XSD<p>Introducing our Java XML Validation Project! Master the art of validating XML effortlessly against XSD in a Java environment. Gain hands-on experience, ensuring the integrity and conformity of your data.</p>
<h2 style="text-align: left;">1. Introduction</h2>
<div>Introducing our Java XML Validation Project! Gain hands-on experience in validating XML documents against XSD with ease. This project serves as a practical guide, equipping you with the necessary knowledge and tools to ensure the integrity and conformity of your XML data in a Java environment. Discover the simplicity of our step-by-step approach, which walks you through the entire validation process. Master the art of validating XML effortlessly and unlock a new level of confidence in your data. Join us on this journey as we empower you to harness the power of XML validation in your Java projects. </div>
<h2 style="text-align: left;">2. Generating Java Classes from XSD</h2>
<div>For this exercise, we will be using an XSD from w3schools
<a href="https://www.w3schools.com/xml/schema_example.asp">https://www.w3schools.com/xml/schema_example.asp</a>.
</div>
<div>
<br />
</div>
<div>To be able to generate the Java classes from XSD, we will be using Eclipse Enterprise Edition. IntelliJ can also do it with the Jakarta plugin, but Eclipse performs better, especially when dealing with XJB files.</div>
<div>
<br />
</div>
<div>IntelliJ</div>
<div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHkRxu6LBQJZWmo1beebu74SoLLMAw791mjZr2SOQ4PWqD9G8h25U1W-cjWdGHVwH1mAxc5LI0KUwM9Ktn3APbyh_MPY1UMeOSQqtFPzXExHmz5i7jtpH373_Luej-IFeJaNilYYTvrBXnPD2wby6QjYZ8E3OzNEtxghY6U36mQccfGxFyJfe07UMAIAC9/s2057/intellij-jakarta-plugin.png" style="clear: left; margin-bottom: 1em; margin-right: 1em; text-align: center;">
<img border="0" data-original-height="620" data-original-width="2057" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHkRxu6LBQJZWmo1beebu74SoLLMAw791mjZr2SOQ4PWqD9G8h25U1W-cjWdGHVwH1mAxc5LI0KUwM9Ktn3APbyh_MPY1UMeOSQqtFPzXExHmz5i7jtpH373_Luej-IFeJaNilYYTvrBXnPD2wby6QjYZ8E3OzNEtxghY6U36mQccfGxFyJfe07UMAIAC9/s16000/intellij-jakarta-plugin.png" />
</a>
</div>
<div>
<br />
</div>
<div>Eclipse JAXB Classes Generator</div>
<div>
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYLWt5MqXoyO37ymQTtriSc9VgT1GpuMA6MyzPeDm54aAiqK8_7DYzBbTvISu-0ZdYC3cbcez5lfPdaeW1HuIWIHRTYn2X3UuSQjIOrFlE4_7hnI8K_Hs3dBCJ0r9BbcXv4OBPXqmZvqIOBwQUDt9SCY0mLbtolR7zQWtO1XLVEgXHPYGXGiztwXMknf7F/s1121/jaxb-generator.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="1020" data-original-width="1121" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYLWt5MqXoyO37ymQTtriSc9VgT1GpuMA6MyzPeDm54aAiqK8_7DYzBbTvISu-0ZdYC3cbcez5lfPdaeW1HuIWIHRTYn2X3UuSQjIOrFlE4_7hnI8K_Hs3dBCJ0r9BbcXv4OBPXqmZvqIOBwQUDt9SCY0mLbtolR7zQWtO1XLVEgXHPYGXGiztwXMknf7F/s16000/jaxb-generator.png" />
</a>
</div>
<br />
<div>Generated Classes</div>
<div>
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUZeneM8cF4-qRmm75X9gU5_Jk3hX6dIXr5wPnXqd8_lhCzy3lBPyIpzToJU-N2yxgO9a9VFrDoolMdBUA3UIqpgABMi90P9MhBAdwo9BfnSaKBXVIcy-ahwMyzliyAoj1gRDw8F0SGcgXSCEdt8taAOnZ5BWNbaeCs8KK5pmnxQd94es2vUUB0gAw0UAS/s2269/generate-java-from-xsd.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="1440" data-original-width="2269" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUZeneM8cF4-qRmm75X9gU5_Jk3hX6dIXr5wPnXqd8_lhCzy3lBPyIpzToJU-N2yxgO9a9VFrDoolMdBUA3UIqpgABMi90P9MhBAdwo9BfnSaKBXVIcy-ahwMyzliyAoj1gRDw8F0SGcgXSCEdt8taAOnZ5BWNbaeCs8KK5pmnxQd94es2vUUB0gAw0UAS/s16000/generate-java-from-xsd.png" />
</a>
</div>
<div><br />
</div>
<h2 style="text-align: left;">3. Dependencies</h2>
<pre class="brush: java" style="text-align: left;"><dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-xml</artifactId>
</dependency>
<dependency>
<groupId>jakarta.activation</groupId>
<artifactId>jakarta.activation-api</artifactId>
<version>2.1.1</version>
</dependency>
<dependency>
<groupId>jakarta.xml.bind</groupId>
<artifactId>jakarta.xml.bind-api</artifactId>
<version>${jakarta.version}</version>
</dependency>
<dependency>
<groupId>org.glassfish.jaxb</groupId>
<artifactId>jaxb-runtime</artifactId>
<version>${jakarta.version}</version>
<scope>runtime</scope>
</dependency>
</pre>
<h2 style="text-align: left;">4. Core Classes</h2>
<div>To run the validations and throw the appropriate exception or name of the tag where a constraint is violated, we will need the following classes.</div>
<h3 style="text-align: left;">4.1 ResourceResolver</h3>
<div>To import an external XSD, for example, if we will be using the XML digital signature.</div>
<pre class="brush: java"><xs:import namespace="http://www.w3.org/2000/09/xmldsig#" schemaLocation="xmldsig-core-schema.xsd"/>
</pre>
<pre class="brush: java" style="text-align: left;">public class ResourceResolver implements LSResourceResolver {
private static final Logger LOGGER = LoggerFactory.getLogger(ResourceResolver.class);
public LSInput resolveResource(String type, String namespaceURI, String publicId, String systemId, String baseURI) {
// note: in this sample, the XSD's are expected to be in the root of the classpath
InputStream resourceAsStream = this.getClass().getClassLoader().getResourceAsStream("xsd/" + systemId);
return new Input(publicId, systemId, resourceAsStream);
}
static class Input implements LSInput {
private String publicId;
private String systemId;
public String getStringData() {
String textDataFromFile;
try {
BufferedReader bufferReader = new BufferedReader(new InputStreamReader(inputStream));
StringBuilder stringBuilder = new StringBuilder();
String eachStringLine;
while ((eachStringLine = bufferReader.readLine()) != null) {
stringBuilder.append(eachStringLine).append("\n");
}
textDataFromFile = stringBuilder.toString();
return textDataFromFile;
} catch (IOException e) {
LOGGER.error("Fail reading from inputStream={}", e.getMessage());
return null;
}
}
...
</pre>
<h3 style="text-align: left;">4.2 SaxErrorHandler</h3>
<div>For us to be able to know the tag where a constraint exception is thrown, we need to define a custom exception that accepts an XMLStreamReader object where we can get the localName property.</div>
<pre class="brush: java" style="text-align: left;">public class SaxErrorHandler implements ErrorHandler {
private XMLStreamReader reader;
public SaxErrorHandler(XMLStreamReader reader) {
this.reader = reader;
}
@Override
public void error(SAXParseException e) {
warning(e);
}
@Override
public void fatalError(SAXParseException e) {
warning(e);
}
@Override
public void warning(SAXParseException e) {
throw new XmlValidationException(reader.getLocalName());
}
}
</pre>
<h3 style="text-align: left;">4.3 Miscellaneous Classes</h3>
<div>We will also be needing utility classes for:</div>
<div>
<ul style="text-align: left;">
<li>Creating a JAXBElement</li>
<li>Convert JAXBElement to XMLStreamReader</li>
<li>Defining the correct marshaller object</li>
</ul>
<div>These classes are all available in the project.</div>
</div>
<h3 style="text-align: left;">4.4 XmlSchemaValidator Interface</h3>
<div>And finally, the interface that pieces together all components to do the validation.</div>
<pre class="brush: java" style="text-align: left;"> default void validateXMLSchema(String xsdPath, JAXBElement<?> xml) {
Validator validator = null;
try {
SchemaFactory factory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
factory.setProperty(XMLConstants.ACCESS_EXTERNAL_DTD, "");
factory.setProperty(XMLConstants.ACCESS_EXTERNAL_SCHEMA, "");
factory.setResourceResolver(new ResourceResolver());
Schema schema = factory.newSchema(ResourceUtils.getFile(xsdPath));
validator = schema.newValidator();
XMLStreamReader xmlStreamReader = asStreamReader(xml);
ErrorHandler errorHandler = new SaxErrorHandler(xmlStreamReader);
validator.setErrorHandler(errorHandler);
validator.validate(new StAXSource(xmlStreamReader));
} catch (IOException | SAXException | JAXBException | XMLStreamException e) {
var xmlEx = getXmlValidationExceptionIfPresent(e);
throw new InvalidXmlException(xmlEx.getLocalName(), e);
}
}
</pre>
<h2 style="text-align: left;">5. Source Code / Git Repository</h2>
<div>The complete source code for this project is available for my sponsors at
<a href="https://github.com/czetsuyatech/java-xml-validation-by-xsd">https://github.com/czetsuyatech/java-xml-validation-by-xsd</a>.
</div>
<div>
<br />
</div>
<div>Sponsorship link:
<a href="https://github.com/sponsors/czetsuya">https://github.com/sponsors/czetsuya</a>
</div>
<div>
<br />
</div>
<div>Project Screenshot</div>
<div>
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgW_ER_T4HVe-1lyK6mJ0bP8Imwjb5-43WgONpjlb93NDirVK8-kt3KgrY4WTuz4v07XvVlaY-bbmFPAWb6A9MRStv5z8F0J6PSofnWvXEbRGp-41kAmMefJ9rf9OnybpJ5AvdTrknSG_t0PNMSPRp8wug1V4i_jwYfafAS2k83QCef0Rkc33D0feLawcYL/s2463/Xml-vaildation-using-xsd.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="1746" data-original-width="2463" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgW_ER_T4HVe-1lyK6mJ0bP8Imwjb5-43WgONpjlb93NDirVK8-kt3KgrY4WTuz4v07XvVlaY-bbmFPAWb6A9MRStv5z8F0J6PSofnWvXEbRGp-41kAmMefJ9rf9OnybpJ5AvdTrknSG_t0PNMSPRp8wug1V4i_jwYfafAS2k83QCef0Rkc33D0feLawcYL/s16000/Xml-vaildation-using-xsd.png" />
</a>
</div>
czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-2051174956964689072023-05-19T12:55:00.144+08:002023-08-25T17:32:46.263+08:00Efficient HTTP Request Logging in Spring with Zalando: A Developer's Guide<p> Logging internal HTTP requests using RestTemplate with Zalando library.</p><h2 style="text-align: left;">1. Overview</h2><p>In some cases, it is critical that we log both internal and external HTTP requests. This is very useful for debugging communication between microservices. But this feature is already available with other Spring boot libraries. What I wanted to show in this exercise is the flexibility of logging or saving in a database of particular data.</p><h2 style="text-align: left;">2. Prerequisites</h2><h3 style="text-align: left;">2.1 Dependencies</h3><p>For this exercise, I will use the following dependencies:</p><p></p><ul style="text-align: left;"><li>Spring Boot 3</li><li>logbook-spring-boot-starter v3.x</li></ul><h3 style="text-align: left;">2.2 The Spring Project</h3><div>The project that we will work on will run on Spring Boot 3.x.</div><h2 style="text-align: left;">3. Hands On</h2><h3 style="text-align: left;">3.1 Controller</h3><div>We will start with our controller.</div><p></p>
<pre class="brush: java">@RestController
@Slf4j
@RequiredArgsConstructor
public class LoggingController {
private final RestTemplate restTemplate;
@GetMapping("/logs/internal-requests")
public void logBody() {
log.debug("Logging internal request");
restTemplate.getForEntity("http://localhost:8080/call-me", Computer.class);
}
@GetMapping("/call-me")
public Computer callMe() {
log.debug("calling internal url");
return Computer.builder()
.brand("Apple")
.model("Macbook Pro")
.cpu("M2")
.ram(32)
.build();
}
}
</pre>
<p>
This class exposes a REST endpoint /logs/internal-requests that will trigger an HTTP request to another endpoint /call-me. This call will be intercepted by the logbook.</p><p>Let's see how it works.</p><h3 style="text-align: left;">3.2 Web Configuration</h3><p>We need to create the following beans for the logbook to intercept the HTTP requests.</p>
<pre class="brush: java" style="text-align: left;">@Configuration
@RequiredArgsConstructor
public class WebBeansConfig {
private final Logbook logbook;
@Bean
public Logbook getLogbook() {
return Logbook.builder()
.strategy(new StatusAtLeastStrategy(200))
.sink(new LogSink(new LogFormatter(), new LogWriter()))
.build();
}
@Bean
public RestTemplate restTemplate() {
RestTemplate restTemplate = new RestTemplate();
restTemplate.getInterceptors().add(getLogbookInterceptor());
return restTemplate;
}
private LogbookClientHttpRequestInterceptor getLogbookInterceptor() {
return new LogbookClientHttpRequestInterceptor(logbook);
}
}
</pre>
<h3 style="text-align: left;">3.3 Logbook and Logging Classes</h3><div>Here are the skeletons of the classes needed by the logbook.</div>
<p>After the logbook intercepts the request, it will cause this class to perform pre and post request data manipulation.</p>
<pre class="brush: java">@RequiredArgsConstructor
public class LogSink implements Sink {
private final LogFormatter formatter;
private final LogWriter writer;
@Override
public void write(Precorrelation preCorrelation, HttpRequest request) throws IOException {
}
@Override
public void write(Correlation correlation, HttpRequest request, HttpResponse response) throws IOException {
}
@Override
public void writeBoth(Correlation correlation, HttpRequest request, HttpResponse response) throws IOException {
writer.write(formatter.format(correlation.getId(), request, response));
}
}
</pre>
<p>This class extracts the necessary values from the HTTP request and response object. We can also deduce whether the request is incoming or outgoing.</p>
<pre class="brush: java">// Customizeable message model
public class LogFormatter {
public Message format(String correlationId, HttpRequest request, HttpResponse response) throws IOException {
return Message.builder()
.correlationId(correlationId)
.requestBody(request.getBodyAsString())
.responseBody(response.getBodyAsString())
.build();
}
}
</pre>
<p>Finally, this class logs the message object in the console. We can also save the Message object in the database by injecting a mapper and converting the Message to an entity, then inject a repository that we can use to insert the entity in the database.</p>
<pre class="brush: java" style="text-align: left;">// Log writer
@Slf4j
public class LogWriter {
// you can autowire a service here
public void write(Message message) {
log.debug("Http intercepted message={}", message);
}
}
</pre>
<h2 style="text-align: left;">4. GitHub Repository</h2><p></p><ul style="text-align: left;"><li>Sponsor this blog: <a href="https://github.com/sponsors/czetsuya">https://github.com/sponsors/czetsuya</a></li><li><a href="https://github.com/czetsuyatech/logging-http-zalando/">https://github.com/czetsuyatech/logging-http-zalando/</a></li></ul><p></p>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-66962295714020682032023-05-19T12:53:00.494+08:002023-12-01T13:27:30.923+08:00Unlocking Google Calendar API with Keycloak Via Token Exchange: A Developer's Guide <div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUi2cki8VoCskVhILbDLCGLThqC0XMg_HkiKKcm0WMsGVwLBT3U8LV2ak9ztDC_pSYBob81OjsvfWThZDFcxULcvsyMjB50sge-L1kUKjkVXGXJC3rztU6-Gdz3GzN3tGe8bdG2D_KHwdZKTkTfQ6r6zEPqAizbQMlUa7JiD98_MrSwTpXp1bka64rhQ/s3872/eric-rothermel-FoKO4DpXamQ-unsplash.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="2592" data-original-width="3872" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUi2cki8VoCskVhILbDLCGLThqC0XMg_HkiKKcm0WMsGVwLBT3U8LV2ak9ztDC_pSYBob81OjsvfWThZDFcxULcvsyMjB50sge-L1kUKjkVXGXJC3rztU6-Gdz3GzN3tGe8bdG2D_KHwdZKTkTfQ6r6zEPqAizbQMlUa7JiD98_MrSwTpXp1bka64rhQ/s16000/eric-rothermel-FoKO4DpXamQ-unsplash.jpg" /></a></div><h2 style="text-align: left;">1. Overview</h2>
<p>Unlock the power of Keycloak for seamless authentication and authorization in your applications. This comprehensive guide is designed for software developers who want to integrate Keycloak into their systems to enable secure user login and token exchange with Google. </p>
<p>By following step-by-step instructions and best practices, you'll learn how to leverage Keycloak's robust features to facilitate seamless authentication and obtain a Google token. With this token, you can effortlessly access the Google Calendar API and unlock a wealth of functionality for your application. Enhance user experience and streamline calendar integration - start implementing Keycloak and Google integration today.</p>
<p>You need prior knowledge with:</p>
<p></p>
<ul style="text-align: left;">
<li>Google</li>
<li>Keycloak</li>
<li>Spring Boot</li>
</ul>
<p></p>
<h2 style="text-align: left;">2. Google</h2>
<div>Before we can do the integration with Keycloak we need to create a project in Google Console, create a credential and obtain OAuth client id and secret.</div>
<div>
<br />
</div>
<div>If you haven't done so, you can create a Google console account at <a href="https://cloud.google.com/cloud-console">https://cloud.google.com/cloud-console</a>.</div>
<h3 style="text-align: left;">2.1 Creating the Google Cloud Project</h3>
<div>At the top left of your Google Cloud Console, click the project and click <b>New Project</b>. </div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6AMZ4qT9K_oXqRi8ZThACUvNPeT-vKOyr-lzirmz1_3pb2yNW6OzSFM5AhspW8AgO1uOs67ylo1wHq-1LfDngcos0FFWUt4aB_KNq_JMPT-yLdtkpwceOq-Kz-o54l_xfwMkxKfXJw3cxfznwPju9WLoV13cz89t-n9IYheCcP8boSw734F014EGNZQ/s2573/1%20-%20create%20google%20cloud%20project.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="685" data-original-width="2573" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6AMZ4qT9K_oXqRi8ZThACUvNPeT-vKOyr-lzirmz1_3pb2yNW6OzSFM5AhspW8AgO1uOs67ylo1wHq-1LfDngcos0FFWUt4aB_KNq_JMPT-yLdtkpwceOq-Kz-o54l_xfwMkxKfXJw3cxfznwPju9WLoV13cz89t-n9IYheCcP8boSw734F014EGNZQ/s16000/1%20-%20create%20google%20cloud%20project.png" />
</a>
</div>
<br />
<div>
<br />
</div>
<h2 style="text-align: left;">
<br />
</h2>
<h2 style="text-align: left;">
<br />
</h2>
<h2 style="text-align: left;">
<br />
</h2>
<h2 style="text-align: left;">
<br />
</h2>
<div>I call this project "Lab".</div>
<h3 style="text-align: left;">2.2 Create OAuth2 Client</h3>
<div>The next step is to create the OAuth2 client credential that will give us the client id and secret. This combination will be used when we set up the Google Identity provider in Keycloak.</div>
<div>
<br />
</div>
<div>With the <b>Lab</b> project selected, find API & Services / Credentials in the menu. </div>
<div>
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiO-qnmJdu8vPm6DK0nnU81vi4-Ta-_cBRUa_Z6avPU14jIjv1YefJG770hwPYjGqmqK0rckvLDqxlANO9gZJtK5LJOwWBVF-O8k2Bhri0E-Ac_1GQ7-5UveuNl5-qbHippiOSs6zJbrKJB1um_0v1qBU5Q7FRoX5EeHx54TbEzz0TntgApx-unCjH25w/s1027/2-%20credentials.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="1027" data-original-width="1018" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiO-qnmJdu8vPm6DK0nnU81vi4-Ta-_cBRUa_Z6avPU14jIjv1YefJG770hwPYjGqmqK0rckvLDqxlANO9gZJtK5LJOwWBVF-O8k2Bhri0E-Ac_1GQ7-5UveuNl5-qbHippiOSs6zJbrKJB1um_0v1qBU5Q7FRoX5EeHx54TbEzz0TntgApx-unCjH25w/s16000/2-%20credentials.png" />
</a>
</div>
<br />
<div>
<br />
</div>
<div style="text-align: left;">Click <b>Create Credentials </b>with the following inputs: </div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">Type: OAuth client ID</div>
<div style="text-align: left;">Application type: Web application</div>
<div style="text-align: left;">Name: Lab</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">There are two important entries in this section:</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">
<b>Authorized Javascript origins</b>
</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">This should be the URL where the request for login will come from.</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">
<b>Authorized redirect URIs</b>
</div>
<div style="text-align: left;">
<b>
<br />
</b>
</div>
<div style="text-align: left;">This entry is a common cause of problems if not properly set up. This is where Google will redirect the user after a successful login.</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">Some common redirect URIs.</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">
<b>Amazon Cognito:</b> https://lab.auth.ap-southeast-1.amazoncognito.com/oauth2/idpresponse
</div>
<div style="text-align: left;">
<b>Postman:</b> https://www.getpostman.com/oauth2/callback
</div>
<div style="text-align: left;">
<b>Keycloak broker: </b>http://localhost:8080/realms/czetsuyatech/broker/google/endpoint
</div>
<div style="text-align: left;">
<b>Keycloak token:</b> http://localhost:8080/realms/czetsuyatech/protocol/openid-connect/token
</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">When you have a redirect issue, this is one of the first things you may want to check.</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">This is how my Lab client looks like.</div>
<div style="text-align: left;">
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUKj6bdQW1ygHFJXaJRtxfwCBgYGnGHhbZtnusa4e1mG5o1-zp5y2Ax0oTmgHWOw5-xghzIS32LQ0IQKWbEBftxAbZgwccW8KPxYZfUHHKBGnQMkjLI6dN2FyyVv8D8ULhg8RzVHlfeCVtHscUlH9jIHTv3KTRaFL_pdsrJUsgqwTLsoJMSFgouRDtNg/s1804/3%20-%20client.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="1804" data-original-width="1224" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUKj6bdQW1ygHFJXaJRtxfwCBgYGnGHhbZtnusa4e1mG5o1-zp5y2Ax0oTmgHWOw5-xghzIS32LQ0IQKWbEBftxAbZgwccW8KPxYZfUHHKBGnQMkjLI6dN2FyyVv8D8ULhg8RzVHlfeCVtHscUlH9jIHTv3KTRaFL_pdsrJUsgqwTLsoJMSFgouRDtNg/s16000/3%20-%20client.png" />
</a>
</div>
<br />
<div style="text-align: left;">
<br />
</div>
<h2 style="text-align: left;">
<br />
</h2>
<h2 style="text-align: left;">
<br />
</h2>
<h2 style="text-align: left;">
<br />
</h2>
<div style="text-align: left;">On the same page, you should be able to see the client id and secret. Take note because we will need them later.</div>
<div style="text-align: left;">
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVUvcRcv-zMqbq4Svq-g3bGF4ycaRLlBotK4WbCBYlhTYBi8ygW-TqeBehEK9hvd2oTIFyduUK7fDRJkOQa-SLUGj_COo1la4qWx3qXDLXFj_7Z4AjwPy_bjaG8qqD1hU0E-fDeNh0iHJIUAqqLO0ovov4cxfdboDyhUw3Arss6THNefKm8-A-rbxR5g/s1589/4%20-%20secret.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="735" data-original-width="1589" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVUvcRcv-zMqbq4Svq-g3bGF4ycaRLlBotK4WbCBYlhTYBi8ygW-TqeBehEK9hvd2oTIFyduUK7fDRJkOQa-SLUGj_COo1la4qWx3qXDLXFj_7Z4AjwPy_bjaG8qqD1hU0E-fDeNh0iHJIUAqqLO0ovov4cxfdboDyhUw3Arss6THNefKm8-A-rbxR5g/s16000/4%20-%20secret.png" />
</a>
</div>
<br />
<div style="text-align: left;">
<br />
</div>
<h2 style="text-align: left;">
<br />
</h2>
<h2 style="text-align: left;">
<br />
</h2>
<h3 style="text-align: left;">2.3 Setup OAuth Consent Screen</h3>
<div>This section will ask for several application-related settings. You can enter values as you see fit, but normally I set the following.</div>
<div>
<br />
</div>
<div>
<b>Non-sensitive scopes:</b>
</div>
<div>
<ul style="text-align: left;">
<li>auth-userinfo-email</li>
<li>auth-userinfo-profile</li>
<li>openid</li>
</ul>
<div>Don't forget to add a test user. We will use it to login later.</div>
</div>
<div>
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhG4TvAimCy5__jQ-r1zOMVGjbZRQluKPRIFvvqCmnfJV9-9TKJ5AEVPyZkOYdhlXZGVlJg4UX445qwnll8BWuVQT7O_pTG-od6Cs8n2DCzt0bSMYOlnkPbVmWC3b3fYkmpAmxXCJ4Du3pu7WkiyqyAhtw75jt8lIeE-jyGb1i4hqizIOGTP1tarLPKJQ/s1174/5%20-%20test%20users.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="519" data-original-width="1174" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhG4TvAimCy5__jQ-r1zOMVGjbZRQluKPRIFvvqCmnfJV9-9TKJ5AEVPyZkOYdhlXZGVlJg4UX445qwnll8BWuVQT7O_pTG-od6Cs8n2DCzt0bSMYOlnkPbVmWC3b3fYkmpAmxXCJ4Du3pu7WkiyqyAhtw75jt8lIeE-jyGb1i4hqizIOGTP1tarLPKJQ/s16000/5%20-%20test%20users.png" />
</a>
</div>
<br />
<div>
<br />
</div>
<h2 style="text-align: left;">3. Keycloak</h2>
<div>For this exercise, we need to create a new Keycloak realm and client. We must enable authentication in the client.</div>
<h3 style="text-align: left;">3.1 Client</h3>
<div>
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8rDouX4miD7WXdMl7Zm3b7sIA9sf9r-MfpOvE1PtqzUxDpnU8l3nf2lTihKJSy0dtiRL80TZrcxuEO1aNZuz8joi4_iBslDwYE-mzdwpzpnm0mFhMpHx-xuzZB20OHjfyt_212IgWdD76EmFlj99qaHlWKkx1ZH__3d8vPhkpSyorW2oiVqoFIxzuEw/s1965/6-%20keycloak%20client.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="721" data-original-width="1965" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8rDouX4miD7WXdMl7Zm3b7sIA9sf9r-MfpOvE1PtqzUxDpnU8l3nf2lTihKJSy0dtiRL80TZrcxuEO1aNZuz8joi4_iBslDwYE-mzdwpzpnm0mFhMpHx-xuzZB20OHjfyt_212IgWdD76EmFlj99qaHlWKkx1ZH__3d8vPhkpSyorW2oiVqoFIxzuEw/s16000/6-%20keycloak%20client.png" />
</a>
</div>
<br />
<div>
<br />
</div>
<h2 style="text-align: left;">
<br />
</h2>
<div style="text-align: left;">
<span style="font-weight: normal;">Enabling authentication will give us access to Keycloak's client and secret, which we will need later for testing. Take note of it.</span>
</div>
<div style="text-align: left;">
<span style="font-weight: normal;">
<br />
</span>
</div>
<div style="text-align: left;">
<span style="font-weight: normal;">We also need to download the Adapter config of our newly created client.</span>
</div>
<div style="text-align: left;">
<span style="font-weight: normal;">
<br />
</span>
</div>
<div style="text-align: left;">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0AVatzom3hQdV1ynaWluHacbXHIebTkjqsSBjcFu5SUw8Rz--hVvSYuh4ASYjSLQMknbZBr7KrHm_nnRHyEUOGlpbfYZLlrieDbEfbVN7wj27YUeDFj2ZNmhIOUbLfATypBpaW6U5boqnzsxEgZ7zXW_6xk2BQD9CL9GzRGgvMsPHbrBWHXbPCbBGXg/s3032/7%20-%20download%20adapter%20config.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="961" data-original-width="3032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0AVatzom3hQdV1ynaWluHacbXHIebTkjqsSBjcFu5SUw8Rz--hVvSYuh4ASYjSLQMknbZBr7KrHm_nnRHyEUOGlpbfYZLlrieDbEfbVN7wj27YUeDFj2ZNmhIOUbLfATypBpaW6U5boqnzsxEgZ7zXW_6xk2BQD9CL9GzRGgvMsPHbrBWHXbPCbBGXg/s16000/7%20-%20download%20adapter%20config.png" />
</a>
</div>
<br />
<span style="font-weight: normal;">
<br />
</span>
</div> The downloaded config should look like this. Save it as we will use it in our Spring Boot application later.
<pre class="brush: java" style="text-align: left;">{
"realm": "czetsuyatech",
"auth-server-url": "http://localhost:8080/",
"ssl-required": "external",
"resource": "web-front",
"credentials": {
"secret": "BxYWD11qCAW6ZybPTy6b9M8ej0thTxxx"
},
"confidential-port": 0
}</pre>
<h3 style="text-align: left;">3.2 Identity Provider</h3>
<div>Identity providers are third-party services that allow users to log in to Keycloak. In our example, we will do an authorization as well by exchanging the Keycloak token to Google to allow us to call Google API services such as Calendar and Gmail.</div>
<div>
<br />
</div>
<div>So login to Keycloak as admin, click <b>Identity Providers</b> on the left side, Click <b>Add Provider</b>, and select <b>Google.</b>
</div>
<div>
<br />
</div>
<div>In the Client ID and secret fields, enter the values that we saved when we were configuring the Google client earlier.</div>
<div>
<br />
</div>
<div>Under <b>Advance settings</b>, <b></b>change the value of Scopes to " <i>email profile openid https://www.googleapis.com/auth/calendar https://www.googleapis.com/auth/calendar.events https://www.googleapis.com/auth/gmail.readonly</i>", without double quotes. This will give us permission to call list endpoints from Calendar and Gmail APIs. </div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiG_jVl5puTUVszwLu1XZtxuCbAmgshi3KoEKjGTiya7sDaLFzW8xYJZp5eRj6fKsiap4xHDuEZiOo5qLzEtMAgWVnv6uiMaOADnlOPMjAJchfnIA1-ffAJCMzWr6R-tNP1saarj98zBPmGJErmo7duccOPGxJcrV4cL1DYIvLmLXv5CB9_s_LK0fL5og/s2057/8%20-%20Google%20identity%20provider.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="1625" data-original-width="2057" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiG_jVl5puTUVszwLu1XZtxuCbAmgshi3KoEKjGTiya7sDaLFzW8xYJZp5eRj6fKsiap4xHDuEZiOo5qLzEtMAgWVnv6uiMaOADnlOPMjAJchfnIA1-ffAJCMzWr6R-tNP1saarj98zBPmGJErmo7duccOPGxJcrV4cL1DYIvLmLXv5CB9_s_LK0fL5og/s16000/8%20-%20Google%20identity%20provider.png" />
</a>
</div>
<br />
<div>
<br />
</div>
<h3 style="text-align: left;">3.3 Permissions</h3>
<div>In this section, we will give our client permission to exchange tokens with the newly created Google identity provider.</div>
<div>
<br />
</div>
<div>On the identity provider page, click the <b>Permissions</b> tab. Toggle <b>Permissions enabled</b> to <b>ON</b>. Under the Permission list, click <b>token-exchange.</b>
</div>
<div>
<b>
<br />
</b>
</div>
<div>Set the Decision strategy to <b>Unanimous</b>. Click Save. </div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhRNe8jZVRH_jdMA53XFmQR76LoVDUUO2dMF-LOs-kgI6KTQqO2BFDb4fA29OJkRN0WSH24ujs2t2mqF8z-0eVSvC-jDgzYh3LV0vz0UXEhdMRb3ck-ZhWlibqhCKBjFtJD4Aspfb0dfObDeBcHzwKmFIic8sy0Bjdg5xLbOX5olP63A4NWjUSLHic6g/s2160/9%20-%20permission.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="1607" data-original-width="2160" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhRNe8jZVRH_jdMA53XFmQR76LoVDUUO2dMF-LOs-kgI6KTQqO2BFDb4fA29OJkRN0WSH24ujs2t2mqF8z-0eVSvC-jDgzYh3LV0vz0UXEhdMRb3ck-ZhWlibqhCKBjFtJD4Aspfb0dfObDeBcHzwKmFIic8sy0Bjdg5xLbOX5olP63A4NWjUSLHic6g/s16000/9%20-%20permission.png" />
</a>
</div>
<br />
<b>
<br />
</b>
</div>
<div style="text-align: left;">
<span style="font-weight: normal;">Next, we need to create the permission scope. At the top-left of the page, click </span>
<b>Client details</b>. It should redirect you to the Authorization tab. Open the <b>Policies</b> sub-tab. Create a new policy with the following values:
</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">Type: Client</div>
<div style="text-align: left;">Name: google-token-exchange</div>
<div style="text-align: left;">Clients: web-front (the Keycloak client we created earlier)</div>
<div style="text-align: left;">Logic: Positive</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbZ1RRP9nHg6irXBBLh_621hwV7N8asGziHQMBZIg6rmLIxvQ_dOmwLoS4p5Qsu5P_4mdjAmUMvLaUugP62BP6MiPINZhOV7GXZF4WoMrJHTqsGcGVfPHppxMasvnmcF6KEfvdsgrlj3Hms5fBfaQAgtJccTiLYyJttUBAWClw4TSdu-pKmlzV_iP4lA/s2151/10%20-%20policy.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="1071" data-original-width="2151" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbZ1RRP9nHg6irXBBLh_621hwV7N8asGziHQMBZIg6rmLIxvQ_dOmwLoS4p5Qsu5P_4mdjAmUMvLaUugP62BP6MiPINZhOV7GXZF4WoMrJHTqsGcGVfPHppxMasvnmcF6KEfvdsgrlj3Hms5fBfaQAgtJccTiLYyJttUBAWClw4TSdu-pKmlzV_iP4lA/s16000/10%20-%20policy.png" />
</a>
</div>
<br />
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">Go back to your Google identity provider's permission page and set the <b>Policies</b> to the one we just created. </div>
<div style="text-align: left;">
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbIxPSZk0i78CtGqzWR2Refx5Ii3fhCb5-mGgk4SWhHAJzYqL4VeeZnZTYClTJEyYYojrVh9U79iNo0SZ-QARZPrlZBKWQwQwEp8xL7rdwAldCmWs6dZRwVN_LPNSwj8pnB3ayyC2wFh06OcRYy1vXqtsag6QAHz1arigyBZg8v__Wmoshwh5itjBqkw/s2174/11%20-%20policies.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="1615" data-original-width="2174" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbIxPSZk0i78CtGqzWR2Refx5Ii3fhCb5-mGgk4SWhHAJzYqL4VeeZnZTYClTJEyYYojrVh9U79iNo0SZ-QARZPrlZBKWQwQwEp8xL7rdwAldCmWs6dZRwVN_LPNSwj8pnB3ayyC2wFh06OcRYy1vXqtsag6QAHz1arigyBZg8v__Wmoshwh5itjBqkw/s16000/11%20-%20policies.png" />
</a>
</div>
<br />
<div style="text-align: left;">
<br />
</div>
<h2 style="text-align: left;">
<br />
</h2>
<div style="text-align: left;">The setup should now be ready to exchange tokens.</div>
<h2 style="text-align: left;">4. Testing</h2>
<div>To test the exchange of tokens between Keycloak and Google, I have created a Postman collection where we can:</div>
<div>
<ul style="text-align: left;">
<li>Authenticate a user</li>
<li>Exchange Keycloak to Google token</li>
</ul>
<div>Before we proceed with the testing, we need to run a Keycloak instance. I have prepared a docker-compose to do that.</div>
</div>
<div>
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieNAYzR8VFEjAuemfdEacXnhVeXkoAMC3cAu38N1Vy3tp1w_wmf1rlLJQJGMNSpjN_g-u9lKB8e84iw-MvTtzMbJ74DrM__mqI9mNLl81ATx65KBbbSoeyLfgOKlzDr_lQyMVGot-_CvSOI3Zp7EkjNX0GoxGNMFOAL3HQ7TV-RDBuAETZz2qh016jhQ/s3334/12%20-%20keycloak%20docker%20compose.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="1189" data-original-width="3334" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieNAYzR8VFEjAuemfdEacXnhVeXkoAMC3cAu38N1Vy3tp1w_wmf1rlLJQJGMNSpjN_g-u9lKB8e84iw-MvTtzMbJ74DrM__mqI9mNLl81ATx65KBbbSoeyLfgOKlzDr_lQyMVGot-_CvSOI3Zp7EkjNX0GoxGNMFOAL3HQ7TV-RDBuAETZz2qh016jhQ/s16000/12%20-%20keycloak%20docker%20compose.png" />
</a>
</div>
<br />
<div>
<br />
</div>
<h2 style="text-align: left;">
<br />
</h2>
<h2 style="text-align: left;">
<br />
</h2>
<div style="text-align: left;">We need to authenticate our user, so create a new <b>POST</b> request in Postman and use OAuth2 authorization. Set the URL to http://localhost:8080/realms/czetsuyatech/protocol/openid-connect/token. </div>
<div style="text-align: left;">
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlUM2BIIWsTxhqXeCqBjtXbMvnb-RJn7oCjc-_hk-RMIFOIERjIC5HrGcR7xWiCRuV8y6WRJAL3iybLvY5yQzhr7GfUkWgjYf1G81RDORqQ51OXX3d2ExfRXISrpMPrwjeUUh4DAuR0bPGSZsHuDeMi7m75Ib-BqA4ubudif1xbQc8-wX0f1OEI1gClA/s2800/13%20-%20postman%20oauth.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="1610" data-original-width="2800" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlUM2BIIWsTxhqXeCqBjtXbMvnb-RJn7oCjc-_hk-RMIFOIERjIC5HrGcR7xWiCRuV8y6WRJAL3iybLvY5yQzhr7GfUkWgjYf1G81RDORqQ51OXX3d2ExfRXISrpMPrwjeUUh4DAuR0bPGSZsHuDeMi7m75Ib-BqA4ubudif1xbQc8-wX0f1OEI1gClA/s16000/13%20-%20postman%20oauth.png" />
</a>
</div>
<br />
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">Be sure to set the client id and secret to the ones we saved earlier when we created the Keycloak client.</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">For the truncated values:</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">
<b>Callback URL:</b> https://www.getpostman.com/oauth2/callback
</div>
<div style="text-align: left;">
<b>Auth URL:</b> http://localhost:8080/realms/czetsuyatech/protocol/openid-connect/auth
</div>
<div style="text-align: left;">
<b>Access Token URL:</b> http://localhost:8080/realms/czetsuyatech/protocol/openid-connect/token
</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">Click <b>Get New Access Token</b>. It should redirect us to a Google login page, use the test email address that you added earlier when creating the Google client. </div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">The next page will show you the scopes that will be permitted for your user. We defined this when we created the Google identity provider earlier.</div>
<div style="text-align: left;">
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhx34r4PO6qea9WbLXIu2I2QovUisu-VoBXuV522XLZZL-Ir3m1XcCmhivgsb4rpiEK4denejgW3g2UhpIg3t3XaJp34oMa6p0-v-glm7KmYI33DF5Lauw2k7EQnYRm5m3Da7sx7HT85kY1wuI3VpzAAjpYg7LMy2JiPNfw7yLDILoMpCVah-Tb024tOQ/s1372/14%20-%20google%20scopes.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="1372" data-original-width="1234" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhx34r4PO6qea9WbLXIu2I2QovUisu-VoBXuV522XLZZL-Ir3m1XcCmhivgsb4rpiEK4denejgW3g2UhpIg3t3XaJp34oMa6p0-v-glm7KmYI33DF5Lauw2k7EQnYRm5m3Da7sx7HT85kY1wuI3VpzAAjpYg7LMy2JiPNfw7yLDILoMpCVah-Tb024tOQ/s16000/14%20-%20google%20scopes.png" />
</a>
</div>
<br />
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">Click <b>Allow</b>. </div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">Copy the access token and paste it on a <b>bearerToken</b> variable. The succeeding requests will have Authorization Type = Bearer and will use this value. </div>
<div style="text-align: left;">
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEish8isAvM6jo2JaKAY5HhADLCkK45AVmzqo6t-gD1HER6LwugstI0hgafbN3PmzBe-jFfoaEEFGEGWlvjD4lHNtMq6cpeL83DJ9iOdk9DcrIy1VswBxNde6oc6e_ZvA2jqbzR0jfWFIKeGnykgB_OC5Lz3PHCb9NSmdoJ5fgYz_dTXxrs7ebryYOO0PQ/s1641/15%20-%20bearer%20token.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="618" data-original-width="1641" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEish8isAvM6jo2JaKAY5HhADLCkK45AVmzqo6t-gD1HER6LwugstI0hgafbN3PmzBe-jFfoaEEFGEGWlvjD4lHNtMq6cpeL83DJ9iOdk9DcrIy1VswBxNde6oc6e_ZvA2jqbzR0jfWFIKeGnykgB_OC5Lz3PHCb9NSmdoJ5fgYz_dTXxrs7ebryYOO0PQ/s16000/15%20-%20bearer%20token.png" />
</a>
</div>
<br />
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">
<br />
</div>
<h2 style="text-align: left;">
<br />
</h2>
<div style="text-align: left;">The Body of the request must have the following URL encoded parameters:</div>
<div style="text-align: left;">
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioXtF-u3ayG0vEGoRfUR8FZPtagvTqXLst3c760V9SCbKTt7OAygBiaZ15U-SqGFBBvr45N3X1XwvfhEdH-k-Mk_YJv8bhmqv8g_IKUVwaSqF6UDFvxDH0oHEJYOmFn8uDcZi-MbkxOkhPGJUyT7BiX1RhevioNsdcz2TBXuGw2iUFKfbN6PtAaNRfCA/s1737/16%20-%20exchange%20token%20endpoint.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="889" data-original-width="1737" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEioXtF-u3ayG0vEGoRfUR8FZPtagvTqXLst3c760V9SCbKTt7OAygBiaZ15U-SqGFBBvr45N3X1XwvfhEdH-k-Mk_YJv8bhmqv8g_IKUVwaSqF6UDFvxDH0oHEJYOmFn8uDcZi-MbkxOkhPGJUyT7BiX1RhevioNsdcz2TBXuGw2iUFKfbN6PtAaNRfCA/s16000/16%20-%20exchange%20token%20endpoint.png" />
</a>
</div>
<br />
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">Click <b>Send</b>. It should return with the following response. </div>
<div style="text-align: left;">
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4SjkkopcVHJztEKYGEIdYH87yDBuNOq-nATebe7k4dBlidgw2QA_rAgmnZr3QDf-BSMZtvG-uorNqYn6T3uCq36GiaqUAfAw_lZK1JIFbqrsSjgSNdye2bK3rHeAt7BrqjRdSjXB6dfK3b4LDfv5BC3hboa6f55ijNeg1HRAWv0QWAVaBlXPX2Lybyw/s2597/17%20-%20exchange%20token%20response.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;">
<img border="0" data-original-height="600" data-original-width="2597" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh4SjkkopcVHJztEKYGEIdYH87yDBuNOq-nATebe7k4dBlidgw2QA_rAgmnZr3QDf-BSMZtvG-uorNqYn6T3uCq36GiaqUAfAw_lZK1JIFbqrsSjgSNdye2bK3rHeAt7BrqjRdSjXB6dfK3b4LDfv5BC3hboa6f55ijNeg1HRAWv0QWAVaBlXPX2Lybyw/s16000/17%20-%20exchange%20token%20response.png" />
</a>
</div>
<br />
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">This is the token that we can use to access Google APIs.</div>
<div style="text-align: left;">
<br />
</div>
<div style="text-align: left;">Obviously, we need to automate the process if we wanted this feature in our application. Thus, I have created a Spring project to do it.</div>
<h2 style="text-align: left;">5. Spring Project</h2>
<div>This project demonstrates the token-exchange feature available on Keycloak. With this, we can exchange Keycloak for Google's token, which we can use to call Google APIs.</div><div><br /></div><div><i><span style="color: red;"><b>Note:</b> This project is only available for my sponsors </span></i><span style="color: red;"><i>https://github.com/sponsors/czetsuya</i></span><i><span style="color: red;">.</span></i></div>
<h3 style="text-align: left;">5.1 Available Features</h3>
<div>
<div>
<ul style="text-align: left;">
<li>Get the bearer token from the request and store it in a Bean that can be autowired.</li>
<li>Exchange Keycloak token to Google, this will allow us to call Google APIs (Calendar, Gmail).</li>
<li>Use Spring exchange to provide abstraction in calling Google APIs.</li>
<li>Endpoints to list Calendar events and Gmail messages.</li>
</ul>
<div>The heart of the application exchanges tokens between Keycloak and Google.</div>
</div>
</div>
<pre class="brush: java" style="text-align: left;">@Component
public class GoogleTokenExchange implements TokenExchange {
@Override
public String exchangeToken(String accessToken) {
AuthzClient authzClient = AuthzClient.create();
Configuration keycloakConfig = authzClient.getConfiguration();
String baseUrl = authzClient.getServerConfiguration().getTokenEndpoint();
RestTemplate restTemplate = new RestTemplate();
HttpHeaders headers = new HttpHeaders();
headers.add("Content-Type", MediaType.APPLICATION_FORM_URLENCODED.toString());
headers.add("Accept", MediaType.APPLICATION_JSON.toString());
MultiValueMap<String, String> requestBody = new LinkedMultiValueMap<>();
requestBody.add("client_id", keycloakConfig.getResource());
requestBody.add("client_secret", keycloakConfig.getCredentials().get("secret").toString());
requestBody.add("grant_type", "urn:ietf:params:oauth:grant-type:token-exchange");
requestBody.add("subject_token_type", "urn:ietf:params:oauth:token-type:access_token");
requestBody.add("subject_token", accessToken);
requestBody.add("requested_issuer", "google");
HttpEntity formEntity = new HttpEntity<>(requestBody, headers);
ResponseEntity<AccessTokenResponse> response = restTemplate.exchange(baseUrl, HttpMethod.POST, formEntity,
AccessTokenResponse.class);
return response.getBody().getToken();
}
}
</pre>
<h3 style="text-align: left;">5.2 Testing</h3><div>If you have subscribed as my <b>sponsor</b>, you will be given access to my repository and proceed with this testing.</div><div><br /></div><div>The project provides two endpoints, to test calling list calendar events and Gmail messages from Google. Matching Postman requests are also available to make testing easier.</div><div><br /></div><div>As we have done earlier, we need to get an access token from Keycloak and set it as bearerToken environment variable inside Postman.</div><div><br /></div><div><b>Get Calendar Events Response</b></div><div><b><br /></b></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-_-Qx4s9KkeTRkfsR0iw9NwMzT-msQpuj52mIVb15hSm74oCfV7TMpyTSIVWIeCFHlqkz90yr7we93I5tqy1yeL5cpQa40FsyQp5jsxNI-D-8nx3tPCrF5e0Y5F-hzVj789HAuVrTOYjw3gmW5Dlu30KD2uQPlupsKi3K__V7RZVjJgv74Khr0lWxLw/s2138/19%20-%20get%20calendar%20entries.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="1742" data-original-width="2138" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-_-Qx4s9KkeTRkfsR0iw9NwMzT-msQpuj52mIVb15hSm74oCfV7TMpyTSIVWIeCFHlqkz90yr7we93I5tqy1yeL5cpQa40FsyQp5jsxNI-D-8nx3tPCrF5e0Y5F-hzVj789HAuVrTOYjw3gmW5Dlu30KD2uQPlupsKi3K__V7RZVjJgv74Khr0lWxLw/s16000/19%20-%20get%20calendar%20entries.png" /></a></div><br /><b><br /></b></div><h2 style="text-align: left;"><br /></h2><div style="text-align: left;"><b>Get Gmail Messages</b></div><div style="text-align: left;"><b><br /></b></div><div style="text-align: left;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8xeI2Wob9ek19ohSghnoNdcar8sYue-7GcPmXyYNgIpmOA5QmWglkPyna7UC5lPhuIC9y9_F3WR2E2m1P8vqj2oZJmkhWF35CzLqHuvdWVtsmePww-ZNP37jUAh2Rec7ghdHdbM0fHUbBwtEKMEovzWw_tnXBbYAZNfLypCpSooL1XKe1sPzqdqYQfg/s2030/20%20-%20get%20email%20messages.png" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="1850" data-original-width="2030" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8xeI2Wob9ek19ohSghnoNdcar8sYue-7GcPmXyYNgIpmOA5QmWglkPyna7UC5lPhuIC9y9_F3WR2E2m1P8vqj2oZJmkhWF35CzLqHuvdWVtsmePww-ZNP37jUAh2Rec7ghdHdbM0fHUbBwtEKMEovzWw_tnXBbYAZNfLypCpSooL1XKe1sPzqdqYQfg/s16000/20%20-%20get%20email%20messages.png" /></a></div><br /><b><br /></b></div><h2 style="text-align: left;">6. Source Code</h2>
<div>All the resources used in this article and demo are available on the GitHub repository https://github.com/czetsuyatech/auth-exchange.</div>
<div>
<ul style="text-align: left;">
<li>Spring project source code</li>
<li>Docker/compose file</li>
<li>Keycloak realm</li>
<li>Postman collection for testing</li>
</ul>
</div>
<h2 style="text-align: left;">7. References</h2>
<div>
<ul style="text-align: left;">
<li>https://www.keycloak.org/docs/latest/securing_apps/#_token-exchange</li><li>https://www.keycloak.org/docs/latest/server_development/index.html#retrieving-external-idp-tokens</li><li>https://developers.google.com/calendar</li><li>https://developers.google.com/gmail/api/guides</li>
</ul>
</div>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-51196942180825227572023-05-07T07:28:00.007+08:002023-05-20T17:10:49.452+08:00Czetsuya Tech Sponsorship Benefits<h3 style="text-align: left;">Bronze</h3><div><ul style="text-align: left;"><li>Sponsorship badge</li><li>NO access to private repositories</li></ul></div><h3 style="text-align: left;">Silver</h3><div><ul style="text-align: left;"><li>You will have access to one of my private repositories</li><li>Does not include the repositories in the higher tiers</li></ul></div><h3 style="text-align: left;">Gold</h3><div><ul style="text-align: left;"><li>You will have access to a set of my selected private repositories</li><ul><li>Pre-Quarkus Custom Keycloak</li><ul><li><a href="https://github.com/czetsuyatech/ct-keycloak-iam">https://github.com/czetsuyatech/ct-keycloak-iam</a></li><li><a href="https://github.com/czetsuyatech/ct-keycloak-spis">https://github.com/czetsuyatech/ct-keycloak-spis</a></li><li><a href="https://github.com/czetsuyatech/ct-aws-cognito-spring-security">https://github.com/czetsuyatech/ct-aws-cognito-spring-security</a></li><li><a href="https://github.com/czetsuyatech/ct-keycloak-spring-security">https://github.com/czetsuyatech/ct-keycloak-spring-security</a></li><li><a href="https://github.com/czetsuyatech/ct-iam-spring-security">https://github.com/czetsuyatech/ct-iam-spring-security</a></li><li><a href="https://github.com/czetsuyatech/java-x509certificate-validation">https://github.com/czetsuyatech/java-x509certificate-validation</a></li><li><a href="https://github.com/czetsuyatech/java-xml-digital-signature">https://github.com/czetsuyatech/java-xml-digital-signature</a></li><li><a href="https://github.com/czetsuyatech/unified-kafka-events">https://github.com/czetsuyatech/unified-kafka-events</a></li><li><a href="https://github.com/czetsuyatech/unified-kafka-events-client">https://github.com/czetsuyatech/unified-kafka-events-client</a></li><li><a href="https://github.com/czetsuyatech/auth-exchange">https://github.com/czetsuyatech/auth-exchange</a></li></ul></ul><li>Does not include the repositories in the higher tiers.</li></ul></div><h3 style="text-align: left;">Diamond</h3><div><ul style="text-align: left;"><li>You will have access to my HiveMaster set of repositories that provides multi-tenancy IAM using Keycloak and Spring.</li><ul><li>Keycloak (>v21.0.2)</li><li><a href="https://github.com/czetsuyatech/hivemaster-iam">https://github.com/czetsuyatech/hivemaster-iam</a></li><li><a href="https://github.com/czetsuyatech/hivemaster-spring-security">https://github.com/czetsuyatech/hivemaster-spring-security</a></li><li><a href="https://github.com/czetsuyatech/hivemaster-spring-client">https://github.com/czetsuyatech/hivemaster-spring-client</a></li><li><a href="https://github.com/czetsuyatech/hivemaster-spis">https://github.com/czetsuyatech/hivemaster-spis</a></li><li><a href="https://github.com/czetsuyatech/hivemaster-keycloak-event-bridge">https://github.com/czetsuyatech/hivemaster-keycloak-event-bridge</a></li><li><a href="https://github.com/czetsuyatech/hivemaster-iam-services">https://github.com/czetsuyatech/hivemaster-iam-services</a></li><li><a href="https://github.com/czetsuyatech/hivemaster-clients-java">https://github.com/czetsuyatech/hivemaster-clients-java</a></li><li><a href="https://github.com/czetsuyatech/hivemaster-api-specs">https://github.com/czetsuyatech/hivemaster-api-specs</a></li></ul></ul><div>For contract services <a href="https://www.czetsuyatech.com/p/consultation-services.html">https://www.czetsuyatech.com/p/consultation-services.html</a>.</div></div><p><i>Keep coding, and keep supporting!</i></p>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-23837438197396164862023-05-06T17:55:00.014+08:002023-05-07T13:12:08.502+08:00Implementing X509 Certificate Validation in Java: A Step-by-Step Guide<p>Unlock the power of secure communication with Java! Learn how to implement X509 certificate validation simply and straightforwardly, step-by-step.</p>
<h2 style="text-align: left;">Overview</h2>
<div>
<div>An X.509 certificate is a digital certificate used to verify a particular entity's identity, such as a website or an individual. X.509 certificates are widely used in various applications, including secure communication protocols like HTTPS, SSL, and TLS.</div>
<div>
<br />
</div>
<div>An X.509 certificate contains several pieces of information, including the identity of the entity being verified, the public key that is associated with that entity, and various other pieces of metadata that provide additional information about the certificate itself.</div>
<h2 style="text-align: left;">Importance</h2>
<div>One of the key benefits of X.509 certificates is that they allow entities to authenticate themselves to other parties securely and reliably. This can help to prevent various types of attacks, including man-in-the-middle attacks, where an attacker intercepts and modifies communication between two parties to steal sensitive information.</div>
<div>
<br />
</div>
<div>Overall, X.509 certificates play a critical role in modern security infrastructure and are essential to many different types of secure communication protocols.</div>
</div>
<h2 style="text-align: left;">Requirements</h2>
<div>
<div>You must have the following installed on your computer.</div>
<div>- Git</div>
<div>- Maven</div>
<div>- IDE (IntellIJ / Visual Studio)</div>
<div>- Java 17+</div>
</div>
<div>
<h2 style="text-align: left;">Dependencies</h2>
<div>This project uses the following dependencies.</div>
<div>- bcpkix-jdk15on - certificate revocation check</div>
<div>- spring-boot-starter-test (for testing)</div>
<div>- lombok (for logging)</div>
<div>- Certificates from https://www.digicert.com/kb/digicert-root-certificates.htm</div>
</div>
<h2 style="text-align: left;">Project Code</h2><div>The certificate validation process.</div><div><br /></div><div><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg4_ReIU4kjr9c7FMgel2sDqj1vWCb1HsQdb_HI94ywi7usn1bsveXpkzZa-lkuvjuSDgYhgE2tsFni-TfVqKy0BP-5yFexsLDfxvpJupMbcg2x7s5Cjnn0ESIgpPiSMxvxULmkYxQqmUT7XetSYmVRhpnnkNXIuKWky7cnswoSbpmAYw8vklqYxOf8Cw" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img data-original-height="438" data-original-width="441" src="https://blogger.googleusercontent.com/img/a/AVvXsEg4_ReIU4kjr9c7FMgel2sDqj1vWCb1HsQdb_HI94ywi7usn1bsveXpkzZa-lkuvjuSDgYhgE2tsFni-TfVqKy0BP-5yFexsLDfxvpJupMbcg2x7s5Cjnn0ESIgpPiSMxvxULmkYxQqmUT7XetSYmVRhpnnkNXIuKWky7cnswoSbpmAYw8vklqYxOf8Cw=s16000" /></a></div>
<div>To implement the validation, we will introduce a bean where we can define the:</div>
<div>
<ol style="text-align: left;">
<li>Certificate that we are validating</li>
<li>The pem file that we can cross-check #1</li>
<li>Root certificate to check if the certificate is already revoked</li></ol>
<div>Let's examine the code.</div>
</div>
<div>The CertificateBundle class is just a simple Java bean.</div>
<pre class="brush: java">@Getter
@Setter
@Builder
public class CertificateBundle {
// Certificate extracted from the message
private X509Certificate certificate;
// Certificate loaded from the application must match the certificate
private X509Certificate confirmationCertificate;
// RootCertificate of the certificate
private X509Certificate rootCertificate;
}
</pre>
<div>To validate the <b>certificate expiration</b>, we need to extract the X509Certificate object and perform a time comparison against the value of getNotAfter(). Here, we're making sure that the current date is greater than the certificate's expiry date.</div>
<pre class="brush: java">Date expiresOn = cert.getNotAfter();
Date now = new Date();
if (now.getTime() > expiresOn.getTime()) {
throw new ExpiredCertificateException();
}
</pre>
<div><b>Comparing digital signature against a PEM file.</b> In most fintech applications, there is a need to sign the message before sending it to another service. Upon receiving the message, the server needs to extract the public certificate embedded in it and compare it to a PEM file. This PEM file is a public key of the private key certificate used to sign the message.</div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiaoKK27UU61MIL6aa4rk0j_rC83fpASrGEELGrYyeVHu4Emd8yzJdVG5Ba4vjbCFei9iZ2ERcXMCrWlSRQQ0EOuVgntpWGznjqs9BIlxZ6R7pX2bR99U19BB4bN0yHf9oqDMuWKNdLGG5I78iXDcE5kCrIbimOv52YGJUFnvUyESmSj0Eqk6HRbZR7ew" style="margin-left: 1em; margin-right: 1em;"><img data-original-height="301" data-original-width="437" src="https://blogger.googleusercontent.com/img/a/AVvXsEiaoKK27UU61MIL6aa4rk0j_rC83fpASrGEELGrYyeVHu4Emd8yzJdVG5Ba4vjbCFei9iZ2ERcXMCrWlSRQQ0EOuVgntpWGznjqs9BIlxZ6R7pX2bR99U19BB4bN0yHf9oqDMuWKNdLGG5I78iXDcE5kCrIbimOv52YGJUFnvUyESmSj0Eqk6HRbZR7ew=s16000" /></a></div><br /></div>
<pre class="brush: java">boolean match = Arrays.equals(cert1.getPublicKey().getEncoded(),
cert2.getPublicKey().getEncoded());
if (!match) {
throw new MismatchCertificateException("Embedded certificate does not match the given PEM");
}
</pre>
<div>And finally, we do the <b>revocation check. </b>This process allows us to know whether a website's certificate is trustworthy or not. It uses the root certificate that we used for signing as the trust anchor.</div>
<pre class="brush: java">try {
List<X509Certificate> certs = Collections.singletonList(cb.getCertificate());
CertificateFactory cf = CertificateFactory.getInstance("X509");
CertPath cp = cf.generateCertPath(certs);
// load the root CA cb
X509Certificate rootCACert = cb.getRootCertificate();
// init trusted certs
TrustAnchor ta = new TrustAnchor(rootCACert, null);
Set<TrustAnchor> trustedCerts = new HashSet<>();
trustedCerts.add(ta);
// init PKIX parameters
PKIXParameters params = new PKIXParameters(trustedCerts);
// load the CRL
params.addCertStore(CertStore.getInstance("Collection",
new CollectionCertStoreParameters(getX509Crls(cf, getCrl(cb.getCertificate())))));
// perform validation
CertPathValidator cpv = CertPathValidator.getInstance("PKIX");
PKIXCertPathValidatorResult cpvResult = (PKIXCertPathValidatorResult) cpv.validate(cp,
params);
X509Certificate trustedCert = cpvResult.getTrustAnchor().getTrustedCert();
if (trustedCert == null) {
log.error("Trusted certificate not found");
throw new CertificateException("Trusted certificate not found");
} else {
log.debug("Trusted CA DN = " + trustedCert.getSubjectX500Principal());
}
} catch (InvalidAlgorithmParameterException | java.security.cert.CertificateException |
CRLException |
NoSuchAlgorithmException |
CertPathValidatorException |
IOException e) {
throw new RevokedCertificateException(e);
}
</pre>
<h2 style="text-align: left;">Repository</h2><div><ul style="text-align: left;"><li><a href="https://github.com/czetsuyatech/java-x509certificate-validation">https://github.com/czetsuyatech/java-x509certificate-validation</a> - Available for my Silver sponsors: <a href="https://github.com/sponsors/czetsuya">https://github.com/sponsors/czetsuya</a>.</li>
</ul>
</div>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-73686693325466874352023-03-31T11:35:00.002+08:002023-03-31T11:35:26.995+08:00How to Use Lob Datatype in PostgreSQL<h2 style="text-align: left;">1. Problem</h2>
<p>Here's the typical code we use when creating a Lob column.</p>
<p>Column definition</p>
<pre class="brush: java">@NotNull
@Lob
@Column(name = "message")
private String message;
</pre>
<p>But when we create our tables we would encounter the error below, because PostgreSQL doesn't support lob.</p>
<pre class="brush: java">Caused by: org.hibernate.tool.schema.spi.SchemaManagementException: Schema-validation: wrong column type encountered in column [message] in table [comm_messages]; found [text (Types#VARCHAR)], but expecting [oid (Types#CLOB)]
at org.hibernate.tool.schema.internal.AbstractSchemaValidator.validateColumnType(AbstractSchemaValidator.java:167)
</pre>
<h2 style="text-align: left;">2. Solution</h2>
<p>To resolve this problem we can implement several solutions.</p>
<p>2.1 Set the column to TEXT in the table. With this approach, we need to manually edit the entity where the column is defined.</p>
<pre class="brush: java">@NotNull
@Lob
@Column(name = "message", columnDefinition = "TEXT")
private String message;
</pre>
<p>2.2 If you are using Spring, you can set the non_contextual_creation to true in the application property file.</p>
<pre class="brush: java">spring:
jpa:
properties:
hibernate:
jdbc:
lob:
non_contextual_creation: true
</pre>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-17013756009492057542023-03-19T20:30:00.006+08:002023-03-19T20:36:08.647+08:00How to Use EntityListener for Effective Audit Trails in Spring<h2 style="text-align: left;">1. Overview</h2>
<div>
<div>An audit trail is important in persisting entities using JPA because it helps maintain data integrity and ensure compliance with regulatory requirements. An audit trail records a history of changes made to data over time, including who made the changes and when. By implementing an audit trail, you can track and verify any changes made to data, and identify any unauthorized or unexpected modifications. This can help detect and prevent data tampering, and provide transparency into data changes made by different users over time.</div>
<div>
<br />
</div>
<div>In the context of JPA, audit trails can be implemented using various techniques, including adding auditable columns to database tables or using interceptors to capture changes made to entities before they are persisted in the database. By including an audit trail in your JPA application, you can maintain a comprehensive history of changes made to data, which can be useful for various purposes such as auditing, compliance, and troubleshooting.</div>
<div>
<br />
</div>
<div>Overall, implementing an audit trail can provide greater transparency and accountability for data changes in your JPA application, which can help ensure data integrity and protect against unauthorized access or data tampering.</div>
</div>
<h2 style="text-align: left;">2. Hands-on Coding</h2>
<div>Spring-data-jpa provides a convenient solution for implementing audit trails in your Java applications, and in this tutorial, we'll show you how to use it to quickly get started with auditing your classes.</div>
<h3 style="text-align: left;">2.1 Auditable Entity</h3>
<div>An abstract class where we will store the audit information such as date and user. It should be annotated with AuditingEntityListener from spring-data-jpa. I'm using Lombok dependency for getter and setter here.</div>
<pre class="brush: java">
@Data
@SuperBuilder
@MappedSuperclass
@NoArgsConstructor
@EntityListeners(AuditingEntityListener.class)
public abstract class AuditableEntity extends BaseEntity {
@Column(name = "created_at", nullable = false, updatable = false)
@CreatedDate
private LocalDateTime createdAt;
@Column(name = "updated_at", nullable = false)
@LastModifiedDate
private LocalDateTime updatedAt;
@Column(name = "created_by", nullable = false, updatable = false)
@CreatedBy
private String createdBy;
@Column(name = "updated_by", nullable = false)
@LastModifiedBy
private String updatedBy;
}
</pre>
<h3 style="text-align: left;">2.2 Auditor Information</h3>
<div>A class where we can get the currently logged user using SecurityContextHolder or a constant for system user.</div>
<pre class="brush: java">
public class AuditorAwareImpl implements AuditorAware<String> {
@Override
public Optional<String> getCurrentAuditor() {
return Optional.of("SYSTEM");
}
}
</pre>
<h3 style="text-align: left;">2.3 Jpa Configuration</h3>
<div>We need to tell Spring context where to pull the auditor information as such we need to add this configuration.</div>
<pre class="brush: java">
@Configuration
@EnableJpaAuditing(auditorAwareRef = "auditorProvider")
@EnableTransactionManagement
public class JpaConfig {
@Bean
public AuditorAware<String> auditorProvider() {
return new AuditorAwareImpl();
}
}
</pre>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-68789821574298388582023-03-19T19:34:00.005+08:002023-03-19T20:33:40.636+08:00A Guide to Implementing Auditing with JPA and Generators<h2 style="text-align: left;">1. Overview</h2>
<p>Auditability refers to the ability to track and record changes made to data in a database over time. This process involves keeping track of information such as who made the changes and when they were made. By tracking this information, it becomes possible to audit the changes and identify any unauthorized or unexpected modifications.</p>
<p>In the context of a database table, auaudibilityypically involves keeping track of the creation and updates of individual rows in the table. This can include information such as the user who created the row, the timestamp of when it was created, and any subsequent updates made to the row along with their corresponding timestamps and user information.</p>
<p>Overall, implementing audiaudibility helps organizations maintain data integrity, comply with regulatory requirements, and provide transparency into data changes made by different users over time.</p>
<h2 style="text-align: left;">2. Hands-on Coding</h2>
<div>For this tutorial, we will need to create a user holder, a user generator, and our auditable entity.</div>
<div>
<br />
</div>
<h3 style="text-align: left;">2.1 User Holder</h3>
<div>A Java class that will hold the user object (String) in a ThreadLocal.</div>
<pre class="brush: java">
public class LoggedUser {
private static final ThreadLocal<String> userHolder = new ThreadLocal<>();
public static void logIn(String user) {
userHolder.set(user);
}
public static void logOut() {
userHolder.remove();
}
public static String get() {
return userHolder.get();
}
}
// How to use this REST. Make sure to get the principal information
@Provider
public class LoggedUserFilter implements ContainerRequestFilter {
@Override
public void filter(ContainerRequestContext requestContext) throws IOException {
LoggedUser.logIn("CRUD");
}
}
</pre>
<h3 style="text-align: left;">2.2 User Generator</h3>
<div>This class will implement the hibernate's ValueGenerator to return our current user.</div>
<pre class="brush: java">
import org.hibernate.Session;
import org.hibernate.tuple.ValueGenerator;
public class LoggedUserGenerator implements ValueGenerator<String> {
@Override
public String generateValue(Session session, Object owner) {
return LoggedUser.get();
}
}
</pre>
<h3 style="text-align: left;">2.3 Auditable Entity</h3>
<div>Finally, to implement auditability in JPA applications, developers can extend this abstract class with pre-defined auditable columns such as createdBy and updatedBy, which simplifies the tracking of changes to data in database tables. I'm using Lombok dependency for the annotations here.</div>
<pre class="brush: java">
@Data
@SuperBuilder
@MappedSuperclass
@NoArgsConstructor
public abstract class AuditableEntity extends BaseEntity {
@CreationTimestamp
@Column(name = "created_at", nullable = false, updatable = false)
private LocalDateTime createdAt;
@UpdateTimestamp
@Column(name = "updated_at", nullable = false)
private LocalDateTime updatedAt;
@GeneratorType(
type = LoggedUserGenerator.class,
when = GenerationTime.INSERT
)
@Column(name = "created_by")
private String createdBy;
@GeneratorType(
type = LoggedUserGenerator.class,
when = GenerationTime.ALWAYS
)
@Column(name = "updated_by")
private String updatedBy;
}
</pre>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-49667599338938243502023-03-06T15:21:00.001+08:002023-03-06T15:21:21.234+08:00How to Create a Spring RestTemplate using HttpClient5<h2 style="text-align: left;">Overview</h2><div>The RestTemplate serves as the primary Spring class for HTTP access on the client side. It operates on a comparable conceptual level to other templates such as JdbcTemplate, JmsTemplate, and those featured in other Spring Framework and portfolio projects.</div><div><h2 style="text-align: left;">Code</h2><pre class="brush: java"> @Bean
public RestTemplate restTemplate() throws Exception {
char[] password = "ca_pass_123".toCharArray();
SSLContext sslContext = SSLContextBuilder.create()
.loadKeyMaterial(keyStore("classpath:truststore.p12", password), password)
.loadTrustMaterial(null, new TrustSelfSignedStrategy()).build();
final SSLConnectionSocketFactory sslSocketFactory = SSLConnectionSocketFactoryBuilder.create()
.setSslContext(sslContext)
.build();
final HttpClientConnectionManager cm = PoolingHttpClientConnectionManagerBuilder.create()
.setSSLSocketFactory(sslSocketFactory)
.setDefaultTlsConfig(TlsConfig.custom()
.setHandshakeTimeout(Timeout.ofSeconds(30))
.setSupportedProtocols(TLS.V_1_3)
.build())
.build();
CloseableHttpClient httpClient = HttpClients.custom()
.setConnectionManager(cm)
.build();
HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory(httpClient);
return new RestTemplate(factory);
}
private KeyStore keyStore(String file, char[] password) throws Exception {
KeyStore keyStore = KeyStore.getInstance(KEY_TYPE);
File key = ResourceUtils.getFile(file);
try (InputStream in = new FileInputStream(key)) {
keyStore.load(in, password);
}
return keyStore;
}
</pre></div>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-25300101533809376262023-03-06T15:00:00.007+08:002023-05-03T10:27:56.692+08:00Mutual TLS with Spring: Ensuring Secure Communication Between Applications<h2 style="text-align: left;">Overview</h2>
<p>mTLS or Mutual Transport Layer Security is a type of mutual authentication where two parties, the client and the server, validate each other certificate. In this manner, the client and the server trust each other, and the connection is verified.</p>
<p>To be able to create lower environments as well as local ones, we need to create and sign our own certificates.</p>
<h2 style="text-align: left;">Digital Certificates</h2>
<p>It is an electronic file composed of a public/private key pair that authenticates the identity of an entity. It facilitates secure online communication and data exchange between two entities. These two files are commonly associated with file extensions:</p>
<p></p>
<ul style="text-align: left;">
<li>key - private</li>
<li>pem - public</li>
</ul>
<p></p>
<h2 style="text-align: left;">Generating the Certificates</h2>
<div>
<div>In the succeeding steps, we will generate 3 certificates:</div>
<div>
<ol style="text-align: left;">
<li>Root</li>
<li>Client</li>
<li>Server</li>
</ol>
</div>
<div>We will use OpenSSL, available at https://sourceforge.net/projects/openssl.</div>
<div>
<br />
</div>
<div>A default password will be used for all the certificates "ca_pass_123".</div>
<h3 style="text-align: left;">Root Certificate - Private Key</h3>
<div>This command will generate a root certificate that we will use to sign both the client and server certificates. We will use the default password.</div>
<div>
<br />
</div>
<div>Output: rootCA.key</div>
</div>
<div>
<pre class="brush: java"> openssl genrsa -aes256 -out rootCA.key
</pre>
</div>
<div>
<h3 style="text-align: left;">Root Certificate - Public Key</h3>
<div>Request a public certificate from the root certificate that we have just created.</div>
<div>
<br />
</div>
<div>It will ask for the root certificate's password and the following information.</div>
</div>
<div>
<ul style="text-align: left;">
<li>Country Name=PH</li>
<li>State/Province=Laguna</li>
<li>Locality=Filipino</li>
<li>Organization Name=Czetsuya Tech</li>
<li>Organizational Unit=IT</li>
<li>Common Name=localhost</li>
<li>Email Address=czetsuya@gmail.com</li>
</ul>
<div>
<div>CN=localhost should be replaced when creating certificates for different environments. It should match the domain name.</div>
<div>
<br />
</div>
<div>This certificate will be valid for 365 days.</div>
<div>
<br />
</div>
<div>Output: rootCA.pem</div>
</div>
<div>
<pre class="brush: java">openssl req -new -x509 -key rootCA.key -sha256 -days 365 -out rootCA.pem</pre>
</div>
</div>
<h3 style="text-align: left;">Server Certificate</h3>
<div>Let's create our server certificate.</div>
<div>
<br />
</div>
<div>Output: server.key</div>
<pre class="brush: java">openssl genrsa -aes256 -out server.key
</pre>
<div>
<br />
</div>
<div>
<div>Request for a CSR with the key as input.</div>
<div>
<br />
</div>
<div>It will ask for the root certificate's password and the following information same as the root.</div>
</div>
<div>
<ul style="text-align: left;">
<li>Country Name=PH</li>
<li>State/Province=Laguna</li>
<li>Locality=Filipino</li>
<li>Organization Name=Czetsuya Tech</li>
<li>Organizational Unit=IT</li>
<li>Common Name=localhost</li>
<li>Email Address=czetsuya@gmail.com</li>
</ul>
<div>It will ask for the server's password; use the default one. It will also ask for a challenge password and an optional company name that you can leave blank.</div>
<div>
<br />
</div>
<div>A CSR contains information that is required when requesting a certificate from a Certificate Authority.</div>
<div>
<br />
</div>
<div>Output: server.csr</div>
</div>
<pre class="brush: java">openssl req -new -sha256 -key server.key -out server.csr
</pre>
<div>
<h3 style="text-align: left;">Sign the Server Certificate</h3>
<div>We will ask a Certificate Authority to sign our server certificate in the following step. We will do this by using the root certificate that we created earlier. It will ask for the root certificate's password.</div>
<div>
<br />
</div>
<div>Outputs: server.pem, rooCA.srl</div>
</div>
<pre class="brush: java">openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out server.pem -days 365 </pre>
<div>
<ol style="text-align: left;">
<li>Client</li>
<li>Server</li>
</ol>
<div>
<div>We will use OpenSSL, available at https://sourceforge.net/projects/openssl.</div>
<div>
<br />
</div>
<div>A default password will be used for all the certificates "ca_pass_123".</div>
</div>
</div>
<div>
<h3 style="text-align: left;">Client Certificate</h3>
<div>Since we will be doing mutual TLS, we also need to create a client certificate. The client will present its certificate, the server will validate it, and vice-versa.</div>
<div>
<br />
</div>
<div>Output: client.key</div>
</div>
<pre class="brush: java">openssl genrsa -aes256 -out client.key
</pre>
<div>
<div>Request for a CSR with the key as input.</div>
<div>
<br />
</div>
<div>It will ask for the root certificate's password and the following information same as the server.</div>
</div>
<div>
<ul style="text-align: left;">
<li>Country Name=PH</li>
<li>State/Province=Laguna</li>
<li>Locality=Filipino</li>
<li>Organization Name=Czetsuya Tech</li>
<li>Organizational Unit=IT</li>
<li>Common Name=localhost</li>
<li>Email Address=czetsuya@gmail.com</li>
</ul>
<div>
<div>It will ask for the client's password; use the default one. It will also ask for a challenge password and an optional company name that you can leave blank.</div>
<div>
<br />
</div>
<div>A CSR contains information that is required when requesting a certificate from a Certificate Authority.</div>
<div>
<br />
</div>
<div>Output: client.csr</div>
</div>
</div>
<pre class="brush: java">openssl req -new -key client.key -out client.csr
</pre>
<h3 style="text-align: left;">Sign the Client Certificate</h3>
<div>We will ask a Certificate Authority to sign the client certificate in the following step. We will do this by using the root certificate that we created earlier. It will ask for the root certificate's password.</div>
<div>
<br />
</div>
<div>Outputs: client.pem, rooCA.srl</div>
<pre class="brush: java">openssl x509 -req -in client.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out client.pem -days 365
</pre>
<h2 style="text-align: left;">mTLS on Spring Boot Application</h2>
<div>Now, it's time to create a Spring project to demonstrate TLS. It is enough to have spring-boot-starter-web dependency for this exercise.</div>
<div>
<br />
</div>
<div>Postman is also needed to perform tests.</div>
<h3 style="text-align: left;">One Way TLS</h3>
<div>We will start with one-way TLS, where the client will verify that the server certificate is valid. Spring Boot will do the heavy lifting through configuration, but we must provide it with our bundled server's private and public keys.</div>
<div>
<br />
</div>
<div>It will ask for the server password, and you will need to specify a new one for export. Remember this password, as we will use it in our Spring application.</div>
<div>
<br />
</div>
<div>Output: keystore.p12</div>
<div>
<br />
</div>
<pre class="brush: java">openssl pkcs12 -export -in server.pem -out keystore.p12 -name server -inkey server.key
</pre>
<div> You can explore its content using a Keystore Explorer. </div>
<div>
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdSv87L8BNR6euGJCJl3ci48WpeFCMNzm3sTmpPRPC4RMxfjBJgT2B7WhLYKm_YJebHbAFE77UxD8pjq7ffIK2cPFQccxrSu2uZYCHIYcvfJV04mv7lRIalQQ_tA8Yoi_a41C9iJAfycaCcty5_1w_GASKFGbc3_sdsCMNcXFyF_0W0HLYlkL4F-CNtw/s1080/keystore-explorer.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="328" data-original-width="1080" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdSv87L8BNR6euGJCJl3ci48WpeFCMNzm3sTmpPRPC4RMxfjBJgT2B7WhLYKm_YJebHbAFE77UxD8pjq7ffIK2cPFQccxrSu2uZYCHIYcvfJV04mv7lRIalQQ_tA8Yoi_a41C9iJAfycaCcty5_1w_GASKFGbc3_sdsCMNcXFyF_0W0HLYlkL4F-CNtw/s16000/keystore-explorer.png" />
</a>
</div>
<br />
<div>
<div>Copy the keystore.p12 file into your Spring project's src/main/resources folder.</div>
<div>
<br />
</div>
<div>Update the application.xml property file.</div>
</div>
<pre class="brush: java">server:
ssl:
enabled: true
key-store: "classpath:keystore.p12"
key-store-password: ca_pass_123
key-store-type: PKCS12
</pre>
<div>Since we import the keys with a password, we must provide it in the Spring configuration.</div>
<div>
<br />
</div>
<div>Let's create a test controller.</div>
<div>
<br />
</div>
<pre class="brush: java">@RestController
public class MtlsController {
@GetMapping("/mtls-test")
public String hello() {
return "Hello mTLS";
}
}
</pre> When we run the application, we should see the following log, indicating that HTTPS is on. <div>
<br />
</div>
<pre class="brush: java">2023-02-21T17:29:25.167+08:00 INFO 21536 --- [ restartedMain] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (https) with context path ''
</pre>
<h3 style="text-align: left;">Testing the Certificate</h3>
<div>If we access the endpoint using HTTP protocol, we will get the following:</div>
<div>
<br />
</div>
<pre class="brush: java">Bad Request
This combination of host and port requires TLS.
</pre> Changing the protocol to HTTPS doesn't solve the issue. <div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlSsCd0nOeg7be2r2f0ooorDtK2Q291B0zG_v8fvkLmH0rW_ZySDtlTywg1WvVLEHCc1nRGHZwAmVuHx72WgmhsdqHAFKZnHYQ_U5tyl_iQcEdqBT3Pe_A_AoysgfD5LdwBQjkORvwRzcztL1hD0f8XjAZQIbvp4sVLr3j3o10Q3l_ZavKwmIg1usyFw/s998/certificate_error.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="677" data-original-width="998" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlSsCd0nOeg7be2r2f0ooorDtK2Q291B0zG_v8fvkLmH0rW_ZySDtlTywg1WvVLEHCc1nRGHZwAmVuHx72WgmhsdqHAFKZnHYQ_U5tyl_iQcEdqBT3Pe_A_AoysgfD5LdwBQjkORvwRzcztL1hD0f8XjAZQIbvp4sVLr3j3o10Q3l_ZavKwmIg1usyFw/s16000/certificate_error.png" />
</a>
</div>
<br />
<div>
<div>The reason is that Postman is unaware that we are a certificate authority and doesn't know our root certificate. Thus, we need to make some changes in Postman.</div>
<div>
<br />
</div>
<div>At the top, click the gear icon and Settings. Open the certificate tab. Enable CA Certificates and select the rootCA.pem file.</div>
</div>
<div>
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXYuedugMQCo5oKFwpbx0GnpcX-2cpQpF1D_MsVWYHwGD8BZ_bNEMnFsmy9g8QAAiL-UD31qn0Q-9vxAA2K_7TKDMGR8qMgjJzFQTRlEUUIHlMBHazAzpIEhoggadyuqJyfsbuTgVM9GmeZhljuvpholOc-XkkP0Liy8vSvhfB71twsuWhP79Yq5h1bQ/s1632/set_ca_certificate.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="478" data-original-width="1632" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXYuedugMQCo5oKFwpbx0GnpcX-2cpQpF1D_MsVWYHwGD8BZ_bNEMnFsmy9g8QAAiL-UD31qn0Q-9vxAA2K_7TKDMGR8qMgjJzFQTRlEUUIHlMBHazAzpIEhoggadyuqJyfsbuTgVM9GmeZhljuvpholOc-XkkP0Liy8vSvhfB71twsuWhP79Yq5h1bQ/s16000/set_ca_certificate.png" />
</a>
</div>
<br />
<div>
<div>Postman should be able to call the endpoint correctly.</div>
<h3 style="text-align: left;">Two Way TLS</h3>
<div>We must make several changes to enable mutual authentication, where the server validates the client's certificate.</div>
<div>
<br />
</div>
<div>First, we must tell the Spring Boot app to validate the client's certificate by simply adding a config server.client-auth=need.</div>
<div>
<br />
</div>
<div>Running the same test in Postman will fail because the server will now look for the client's certificate. Let's add the certificate to Postman. Once again, go back to Settings / Certificates, and under the client certificates section, add our certificate.</div>
</div>
<div>
<ul style="text-align: left;">
<li>host=localhost</li>
<li>port=8080</li>
<li>CRT file=client.pem</li>
<li>Key file=client.key</li>
<li>PassPhrase=ca_pass_123</li>
</ul>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEga9pXtVxs7FEJ_su9LhFsmIbjqCVsV-nzvD3axrPh23PGXTQqgevJdZ72wiSsCLevIpoCDKQ80c0r0fKRNK4UugSzqMXCNx1c1dbWBA5rrtEFOrhbWpIzeDHoGuPQjUfdkrscmmArfz3hlTOYjgSHTP_XyJbmynrfX04uPPZkXtpFv5DSiqIYYE1QapQ/s1652/client_certs.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="894" data-original-width="1652" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEga9pXtVxs7FEJ_su9LhFsmIbjqCVsV-nzvD3axrPh23PGXTQqgevJdZ72wiSsCLevIpoCDKQ80c0r0fKRNK4UugSzqMXCNx1c1dbWBA5rrtEFOrhbWpIzeDHoGuPQjUfdkrscmmArfz3hlTOYjgSHTP_XyJbmynrfX04uPPZkXtpFv5DSiqIYYE1QapQ/s16000/client_certs.png" />
</a>
</div>
<div>
<br />
</div>
<div>However, enabling client certificate authentication introduces a new problem. The application needs a store to validate the client's certificate, which is called a trust store. Similar to the Keystore, we must add this store to our application.</div>
<div>The command will ask to trust the certificate; type 'yes'.</div>
<div>
<br />
</div>
<pre class="brush: java">keytool -import -file rootCA.pem -alias rootCA -keystore truststore.p12
</pre> Update the application.xml file. <div>
<br />
</div>
<pre class="brush: java">server:
ssl:
key-store: "classpath:keystore.p12"
key-store-password: ca_pass_123
key-store-type: PKCS12
enabled: true
key-password: ca_pass_123
client-auth: need
trust-store: "classpath:truststore.p12"
trust-store-password: ca_pass_123
trust-store-type: PKCS12
</pre> And with that, we should be able to successfully call our REST endpoint with mutual authentication. <div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcytxUouL0xYihbFSqKBv2hWDxd1hMbJQ_kGSZmFjQf09rzKob1Q5X8x_nSHhH12rjRAXsNKdIiyXm4SLxyNdr2y4UZ8o9R4G7D1jSgQcdsRvcXV580mB6QmW0qEDAbIbmbiLviw_UvQ1gn9P7IMTpBfKwz1IL98ScuoOHbo2nXawkEHCFKdgyviCdkA/s1402/mtls_test.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="903" data-original-width="1402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcytxUouL0xYihbFSqKBv2hWDxd1hMbJQ_kGSZmFjQf09rzKob1Q5X8x_nSHhH12rjRAXsNKdIiyXm4SLxyNdr2y4UZ8o9R4G7D1jSgQcdsRvcXV580mB6QmW0qEDAbIbmbiLviw_UvQ1gn9P7IMTpBfKwz1IL98ScuoOHbo2nXawkEHCFKdgyviCdkA/s16000/mtls_test.png" />
</a>
</div>
<h2 style="text-align: left;">What's Next?</h2>
</div>
</div>
<div>I plan to build 2 Spring application that communicates with mTLS. Stay tuned!</div>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-60131414209734538542023-02-27T11:47:00.026+08:002023-05-03T10:18:08.541+08:00Java-Based Implementation of Digital Signatures and X509 Certificates for Secure XML Document Processing<h2 style="text-align: left;">Overview</h2>
<div>
<div>XML Signature is a standard in the digital signature that allows the authentication and validation of data being sent as XML objects in web service communication. Java 6 introduces Java XML Digital Signature API, which allows the generation and validation of signatures in an XML message.</div>
<div>
<br />
</div>
<div>According to RFC 2828, a digital signature is a cryptographic value added to a data object to enable the recipient to verify its origin and integrity. The cryptographic digital signature API in JDK 6 is described in detail in a security lesson in the Java Tutorial.</div>
<div>
<br />
</div>
<div>An XML signature is a type of digital signature that has unique properties. It outlines a process and format for generating digital signatures in the XML format and offers various advanced features. For example, it allows for the signing of multiple pieces of data, either in binary or XML format, and it enables the use of various cryptographic signature algorithms.</div>
<div>
<br />
</div>
<div>An XML signature is capable of signing arbitrary data, whether it is in XML or binary format. Furthermore, it can sign only a portion or a subset of an XML document instead of the entire document. Uniform Resource Identifiers (URIs) identify the data to be signed. XML signatures can be categorized as one or more of three types based on their properties.</div>
</div>
<div>
<br />
</div>
<div>
<div>
<ol style="text-align: left;">
<li>A detached signature is a signature that covers data that exists outside of the Signature element. This data may be located in a different document or file or present within the same document as the signature, such as a related element.</li>
<li>An enveloping signature is a signature that covers data that is contained within the Signature element itself. In other words, the data to be signed is enclosed within the signature element and the signature itself.</li>
<li>An enveloped signature is a type of signature that covers data containing the Signature element. This means the signature is applied to the entire document or data, including the signature element.</li>
</ol>
</div>
</div>
<div>
<h2 style="text-align: left;">Java API Architecture</h2>
<div>The JSR 105 Java Community Process program established the Java XML Digital Signature API to encompass all necessary and suggested features of the W3C's XML-Signature Syntax and Processing Recommendation. The API employs the Java Cryptography Service Provider Architecture, which enables the development of service provider implementations. Service providers create an XML mechanism that identifies the implementation's XML-parsing mechanism. In Java SE 6's implementation, Sun's service provider supports the Document Object Model (DOM) mechanism. For additional details on service providers, refer to the XML Digital Signature API overview.</div>
<div>
<br />
</div>
<div>Table 1. Java Packages in the Java XML Digital Signature API</div>
</div>
<div>
<br />
</div>
<div>
<table>
<tbody>
<tr>
<th>Package</th>
<th>Description</th>
</tr>
<tr>
<td>javax.xml.crypto</td>
<td>This package includes commonly used classes for executing cryptographic operations on XML.</td>
</tr>
<tr>
<td>javax.xml.crypto.dom</td>
<td>This package comprises classes that are specific to the Document Object Model (DOM) for the javax.xml.crypto package.</td>
</tr>
<tr>
<td>javax.xml.crypto.dsig</td>
<td>This package consists of classes that portray the fundamental elements specified in the XML digital signature standard. The most important class is XMLSignature, which permits signing and verifying an XML digital signature. The abstract factory class XMLSignatureFactory creates objects that implement these interfaces.</td>
</tr>
<tr>
<td>javax.xml.crypto.dsig.dom</td>
<td>This package comprises classes that are specific to the Document Object Model (DOM) for the javax.xml.crypto.dsig package.</td>
</tr>
<tr>
<td>javax.xml.crypto.dsig.keyinfo</td>
<td>This package includes classes that depict the KeyInfo structures specified in the XML digital signature recommendation. The abstract factory class KeyInfoFactory creates objects that implement these interfaces.</td>
</tr>
<tr>
<td>javax.xml.crypto.dsig.spec</td>
<td>This package contains classes that represent the input parameters for various algorithms such as digest, signature, transform, or canonicalization, which are utilized in the processing of XML signatures. <br />
</td>
</tr>
</tbody>
</table>
</div>
<div>
<h2 style="text-align: left;">XML Signature</h2>
<div>This article will utilize an example of an enveloped XML signature, which means that it is generated over the contents of an XML document, specifically a sample <b>puchase-order.xml</b>. </div>
</div>
<div>
<br />
</div>
<div>
<pre class="brush: java"><?xml version="1.0"?>
<PurchaseOrder PurchaseOrderNumber="99503" OrderDate="1999-10-20">
<Address Type="Shipping">
<Name>Ellen Adams</Name>
<Street>123 Maple Street</Street>
<City>Mill Valley</City>
<State>CA</State>
<Zip>10999</Zip>
<Country>USA</Country>
</Address>
<Address Type="Billing">
<Name>Tai Yee</Name>
<Street>8 Oak Avenue</Street>
<City>Old Town</City>
<State>PA</State>
<Zip>95819</Zip>
<Country>USA</Country>
</Address>
<DeliveryNotes>Please leave packages in shed by driveway.</DeliveryNotes>
<Items>
<Item PartNumber="872-AA">
<ProductName>Lawnmower</ProductName>
<Quantity>1</Quantity>
<USPrice>148.95</USPrice>
<Comment>Confirm this is electric</Comment>
</Item>
<Item PartNumber="926-AA">
<ProductName>Baby Monitor</ProductName>
<Quantity>2</Quantity>
<USPrice>39.98</USPrice>
<ShipDate>1999-05-21</ShipDate>
</Item>
</Items>
</PurchaseOrder></pre>
</div>
<div>
<div>XML Example 1.</div>
<div>
<br />
</div>
<div>After applying the digital signature, our message should look like this.</div>
</div>
<pre class="brush: java"><?xml version="1.0" encoding="UTF-8" standalone="no"?>
<PurchaseOrder OrderDate="1999-10-20" PurchaseOrderNumber="99503">
<Address Type="Shipping">
<Name>Ellen Adams</Name>
<Street>123 Maple Street</Street>
<City>Mill Valley</City>
<State>CA</State>
<Zip>10999</Zip>
<Country>USA</Country>
</Address>
<Address Type="Billing">
<Name>Tai Yee</Name>
<Street>8 Oak Avenue</Street>
<City>Old Town</City>
<State>PA</State>
<Zip>95819</Zip>
<Country>USA</Country>
</Address>
<DeliveryNotes>Please leave packages in shed by driveway.</DeliveryNotes>
<Items>
<Item PartNumber="872-AA">
<ProductName>Lawnmower</ProductName>
<Quantity>1</Quantity>
<USPrice>148.95</USPrice>
<Comment>Confirm this is electric</Comment>
</Item>
<Item PartNumber="926-AA">
<ProductName>Baby Monitor</ProductName>
<Quantity>2</Quantity>
<USPrice>39.98</USPrice>
<ShipDate>1999-05-21</ShipDate>
</Item>
</Items>
<Signature
xmlns="http://www.w3.org/2000/09/xmldsig#">
<SignedInfo>
<CanonicalizationMethod Algorithm="http://www.w3.org/TR/2001/REC-xml-c14n-20010315"/>
<SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/>
<Reference URI="">
<Transforms>
<Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
</Transforms>
<DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
<DigestValue>rKK35PJzAva2B92cG/cJp6J2BF5JUbKPS5ogYNJkgJo=</DigestValue>
</Reference>
</SignedInfo>
<SignatureValue>dJ2yTNLTjE41e/q4GRAAHBjIJY+Ax3Qn00CyDRTtfdF1WzCUkxj9V7hpp/alwA3/NWeifHrJMr0F
ak8g41u90RcKV/9rX3lRxNyOKApAKpTw6CrAM6PTcHzHF/NWiL/At8v51AZYm55UgjjrB3xxHvUn
BidyBUtRuETIrikUD3hSMGD2u9prNhgAxWZ+QmRx2ga171/tnGo1B0JF+aBnup8kgq0I9Po3BSW3
a7gwAbVlV3R/pFIrht8wfOVjyzQgFnB+SCJ9UwdSwPJqfyrWxSGE5em3hjbX4VGTWeyhS56s0lX5
9vkCk6dw9cxIvyUsAVSTxYX5SRib2ekpIe7Oyg==
</SignatureValue>
<KeyInfo>
<X509Data>
<X509SubjectName>
1.2.840.113549.1.9.1=#161e6564776172642e6c6567617370695f65787440612d746f2d62652e636f6d,CN=localhost,OU=EETS,O=Atobe,ST=Lisboa,C=PT
</X509SubjectName>
<X509Certificate>MIIDiTCCAnECFAtZBLeU3vRAzm0tXHf7pdP07psNMA0GCSqGSIb3DQEBCwUAMIGAMQswCQYDVQQG
EwJQVDEPMA0GA1UECAwGTGlzYm9hMQ4wDAYDVQQKDAVBdG9iZTENMAsGA1UECwwERUVUUzESMBAG
A1UEAwwJbG9jYWxob3N0MS0wKwYJKoZIhvcNAQkBFh5lZHdhcmQubGVnYXNwaV9leHRAYS10by1i
ZS5jb20wHhcNMjMwMjIxMTAzNzM0WhcNMjQwMjIxMTAzNzM0WjCBgDELMAkGA1UEBhMCUFQxDzAN
BgNVBAgMBkxpc2JvYTEOMAwGA1UECgwFQXRvYmUxDTALBgNVBAsMBEVFVFMxEjAQBgNVBAMMCWxv
Y2FsaG9zdDEtMCsGCSqGSIb3DQEJARYeZWR3YXJkLmxlZ2FzcGlfZXh0QGEtdG8tYmUuY29tMIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxpnILKjFkUsbaVhsI366Od6tfCFnZafsk0Vi
KW+k2z2bckig2+OhDr88KItst7LWORFRmo6JltHx1TfWtkYhboVEWOhw0/kVPCBqNdtJiiAyrj5l
pElgHW6Js8ECUgiiupoJeqwrDPCUbCF0MsOQX/swCbZMqQsAQrPRiNCiDviZ9V/aIlQEaDb7fDJT
wfM4pmPSPzMXPRghv58EKNiJQIpmQw9H0LYA5iu/kvV6nYC8ROF/4giTGhPr3qYMyR1WCa5EuFOO
NGUE8Hn8l43DMFa/hFxVECyYqw/wQQngvM0QDlli1K2FRyZYT7k/v+mqBiNYZC50hjkVtvO5Yd8m
sQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQAnLELZRFMZ8FbLNuSWOqmrizOo7BkiT7gyNx1t6fMb
gadx8qnCpytbghr8O61QuzpYp+65hi9yDEDBfNDtvmT0olIhJK8tsVOu+857MEL0tNcNmavWI4qH
N0lJ5HZRjTm0VLz86UZdWFJfyk7H+sZGOLucDVc4+BzcpsCiW1Dyy+rdEeLNjwmsLvPP2ol16Mzc
Ij48aO2DLOut/+OaUl/4TNOrBKZq4Ve2F55RQUwvMDX0BLZlmDy3pI3XSho7KKDdNhl1/0AFJWVh
skZwR+fAK/RbU1fB9Obc9/IoSm6KANsJRCqFbfkS2hIhT3cvQs/cmR5ZRjgcFgl5J3sMCUFX
</X509Certificate>
</X509Data>
</KeyInfo>
</Signature>
</PurchaseOrder>
</pre>
<div>
<p>Here is the code that digitally signs the XML document.</p>
<pre class="brush: java" style="text-align: left;">package com.czetsuyatech.dsig.xml;
import com.czetsuyatech.dsig.SignXml;
import com.czetsuyatech.dsig.exceptions.SignatureNotFoundException;
import com.czetsuyatech.dsig.exceptions.XmlSigningException;
import jakarta.xml.bind.JAXBException;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.security.InvalidAlgorithmParameterException;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.UnrecoverableEntryException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import javax.xml.crypto.MarshalException;
import javax.xml.crypto.dsig.CanonicalizationMethod;
import javax.xml.crypto.dsig.DigestMethod;
import javax.xml.crypto.dsig.Reference;
import javax.xml.crypto.dsig.SignatureMethod;
import javax.xml.crypto.dsig.SignedInfo;
import javax.xml.crypto.dsig.Transform;
import javax.xml.crypto.dsig.XMLSignature;
import javax.xml.crypto.dsig.XMLSignatureException;
import javax.xml.crypto.dsig.XMLSignatureFactory;
import javax.xml.crypto.dsig.dom.DOMSignContext;
import javax.xml.crypto.dsig.dom.DOMValidateContext;
import javax.xml.crypto.dsig.keyinfo.KeyInfo;
import javax.xml.crypto.dsig.keyinfo.KeyInfoFactory;
import javax.xml.crypto.dsig.keyinfo.X509Data;
import javax.xml.crypto.dsig.spec.C14NMethodParameterSpec;
import javax.xml.crypto.dsig.spec.TransformParameterSpec;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.w3c.dom.Document;
import org.w3c.dom.NodeList;
import org.xml.sax.SAXException;
public class DigitalSigneeImpl implements XmlTransformer {
private final String keystoreFile;
private final String keystoreFormat;
private final String keystorePassword;
private final String privateKeyPassword;
private final String keyEntryAlias;
private final XMLSignatureFactory signatureFactory;
public DigitalSigneeImpl(String keystoreFile, String keystoreFormat, String keystorePassword,
String privateKeyPassword, String keyEntryAlias) {
this.keystoreFile = keystoreFile;
this.keystoreFormat = keystoreFormat;
this.keystorePassword = keystorePassword;
this.privateKeyPassword = privateKeyPassword;
this.keyEntryAlias = keyEntryAlias;
// Instantiate a DOM XMLSignatureFactory object to produce the enveloped signature
signatureFactory = XMLSignatureFactory.getInstance("DOM");
}
public Document sign(SignXml signXml) throws XmlSigningException {
// STEP 1
// Create a Reference to the document being signed by specifying an empty URI value to represent the entire
// document. Additionally, specify the SHA256 digest algorithm and use the ENVELOPED Transform
Reference ref;
try {
ref = signatureFactory.newReference("", signatureFactory.newDigestMethod(DigestMethod.SHA256, null),
Collections.singletonList(signatureFactory.newTransform(Transform.ENVELOPED, (TransformParameterSpec) null)),
null, null);
// Create the SignedInfo
SignedInfo signedInfo =
signatureFactory.newSignedInfo(signatureFactory.newCanonicalizationMethod(CanonicalizationMethod.INCLUSIVE,
(C14NMethodParameterSpec) null),
signatureFactory.newSignatureMethod(SignatureMethod.RSA_SHA256, null), Collections.singletonList(ref));
// STEP 2
// Load the KeyStore and retrieve the signing key and certificate
KeyStore ks = KeyStore.getInstance(keystoreFormat);
ks.load(getFileFromResourceAsStream(keystoreFile), keystorePassword.toCharArray());
KeyStore.PrivateKeyEntry keyEntry =
(KeyStore.PrivateKeyEntry) ks.getEntry(keyEntryAlias,
new KeyStore.PasswordProtection(privateKeyPassword.toCharArray()));
X509Certificate cert = (X509Certificate) keyEntry.getCertificate();
// Create the KeyInfo containing the X509Data
KeyInfoFactory kif = signatureFactory.getKeyInfoFactory();
List<Object> x509Content = new ArrayList<>();
x509Content.add(cert.getSubjectX500Principal().getName());
x509Content.add(cert);
X509Data xd = kif.newX509Data(x509Content);
KeyInfo ki = kif.newKeyInfo(Collections.singletonList(xd));
// STEP 3
// Create an instance of the document that will be signed
DocumentBuilderFactory dbf = getDocumentBuilderFactory();
dbf.setNamespaceAware(true);
//TO IMPL , temp change to be compilable
Document doc = dbf.newDocumentBuilder().parse(getInputStream(signXml));
// Initialize a DOMSignContext by providing the RSA PrivateKey and identifying the parent element of the resulting
// XMLSignature
DOMSignContext dsc = new DOMSignContext(keyEntry.getPrivateKey(), doc.getDocumentElement());
// Instantiate the XMLSignature object without signing it yet
XMLSignature signature = signatureFactory.newXMLSignature(signedInfo, ki);
// Marshal, generate, and sign the enveloped signature
signature.sign(dsc);
return doc;
} catch (NoSuchAlgorithmException | InvalidAlgorithmParameterException | ParserConfigurationException |
MarshalException | XMLSignatureException | UnrecoverableEntryException | KeyStoreException |
IOException | CertificateException | SAXException | JAXBException e) {
throw new XmlSigningException(e);
}
}
public boolean validate(Document doc) throws MarshalException, XMLSignatureException {
// Find Signature element
NodeList nl = doc.getElementsByTagNameNS(XMLSignature.XMLNS, "Signature");
if (nl.getLength() == 0) {
throw new SignatureNotFoundException("Signature element not found");
}
// Create a DOMValidateContext and specify a KeySelector and document context
DOMValidateContext valContext = new DOMValidateContext(new X509KeySelector(), nl.item(0));
valContext.setProperty("org.jcp.xml.dsig.secureValidation", Boolean.TRUE);
// Unmarshal the signature
XMLSignature signature = signatureFactory.unmarshalXMLSignature(valContext);
// Validate the signature
boolean result = signature.validate(valContext);
if (!result) {
System.out.println("Signature failed core validation");
boolean sv = signature.getSignatureValue().validate(valContext);
System.out.println("signature validation status: {}" + sv);
if (!sv) {
// Check the validation status of each Reference.
for (Reference reference : signature.getSignedInfo().getReferences()) {
boolean refValid = reference.validate(valContext);
System.out.println("ref[{}] validity status: " + refValid);
}
} else {
System.out.println("Signature passed core validation");
}
} else {
System.out.println("Signature validation OK");
}
return result;
}
/**
* Write an XML document root node to a file
*
* @param doc XML Document
* @param outFile output filename
* @throws FileNotFoundException when input file is not found
* @throws TransformerException when an error occurred in the transformation
*/
public void writeToFile(Document doc, String outFile) throws IOException, TransformerException {
try (OutputStream os = new FileOutputStream(outFile)) {
TransformerFactory tf = getTransformerFactory();
Transformer trans = tf.newTransformer();
trans.transform(new DOMSource(doc), new StreamResult(os));
}
}
}
</pre>
Note that we need the following dependencies.
<pre class="brush: java">
<dependency>
<groupId>jakarta.activation</groupId>
<artifactId>jakarta.activation-api</artifactId>
<version>2.1.1</version>
</dependency>
<dependency>
<groupId>jakarta.xml.bind</groupId>
<artifactId>jakarta.xml.bind-api</artifactId>
<version>${jaxb.version}</version>
</dependency>
<dependency>
<groupId>com.sun.xml.bind</groupId>
<artifactId>jaxb-impl</artifactId>
<version>${jaxb.version}</version>
<scope>runtime</scope>
</dependency>
</pre>
</div>
<h2 style="text-align: left;">For Sponsors</h2>
<div>
<ul style="text-align: left;">
<li>Complete source code with unit tests <a href="https://github.com/czetsuyatech/java-xml-digital-signature">https://github.com/java-czetsuyatech/xml-digital-signature</a></li>
<li>The custom class <b>X509KeySelector</b>. </li>
</ul>
</div>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-71941161676680729772023-02-12T16:42:00.005+08:002023-02-24T15:32:48.947+08:00How to Insert Records in Batch with Spring Data<h2 style="text-align: left;">1. Introduction</h2>
<p>When working with a REST API, we often deal with a single entity to create, update, delete, list, and get. But there are also times when we need to manage entities in batches, and this mostly happens in the back office. For example, in an ecommerce platform, promotions are governed by uploading a CSV file in the backend. And this CSV file contains a list of records that needs to be processed. </p>
<p>Let's see how we can process a list of promotions.</p>
<h2 style="text-align: left;">2. Solutions for Saving a List of Promotions</h2>
<div>For the succeeding examples, let's assume that we have a BaseEntity class that is being extended by PromotionEntity.</div>
<div>
<br />
</div>
<h3 style="text-align: left;">2.1 Reading, Processing, and Saving One Promotion at a Time</h3>
<div>Not the ideal solution, but I still see this approach in production.</div>
<div>
<br />
</div>
<div>
<b>BaseEntity</b>
</div>
<pre class="brush: java">@Getter
@Setter
@SuperBuilder
@NoArgsConstructor
@AllArgsConstructor
@MappedSuperclass
@JsonInclude(JsonInclude.Include.NON_NULL)
public abstract class BaseEntity implements Serializable {
@Serial
private static final long serialVersionUID = 3986494663579679129L;
public static final int NB_PRECISION = 23;
public static final int NB_SCALE = 12;
@Id
@GeneratedValue(strategy = GenerationType.IDENTITY)
@Column(name = "id")
private Long id;
@Version
@Column(name = "version")
private Integer version;
}
</pre>
<div>
<p>And in the service layer, we process the record one by one:</p>
<pre class="brush: java"> @Transactional(rollbackOn = {SQLException.class})
public void createPromotions(List<Promotion> promotions) {
promotionRepository.saveAll(promotions.stream()
.map(service2EntityMapper::asPromotionEntity)
.toList());
}
</pre>
<p></p>
<ul style="text-align: left;">
<li>Line 1 - it's always a good idea to wrap these kinds of operations (create/delete/update) in a transaction</li>
<li>Line 3 - we stream the list of promotions</li>
<li>Line 4 - we convert each promotion POJO to an entity</li>
<li>Line 5 - we collect the promotion entities</li>
<li>Line 3 - we save a list of promotion entities</li>
</ul>
<p></p>
<p>Note that the saveAll(List<E>) method does not necessarily save records in batch, as it only calls save(E) underneath. </p>
<p>We need the bare minimum configuration for this setup for the data source.</p>
<pre class="brush: java">spring:
application:
name: czetsuyatech
datasource:
url: jdbc:h2:mem:testdb
driverClassName: org.h2.Driver
username: sa
password:
</pre>
</div>
<h4 style="text-align: left;">Summary</h4>
<div>
<ul style="text-align: left;">
<li>It's always a good idea to wrap the create/update/delete operations in a transaction</li>
<li>Use stream when dealing with lists of entities</li>
<li>Use DTOs and POJOs when passing data from one layer to another</li>
<li>For table id, use GenerationType.IDENTITY, when dealing with simple requirements</li>
<li>I would use this approach when I need to process each record in a different transaction. In that case, I need to annotate this method with Transaction.NEVER and call another service that would actually do the data insertion. This means it's possible to only save a fraction of the list due to errors.</li>
</ul>
</div>
<h3 style="text-align: left;">2.2 Reading, Processing, and Saving Promotions in Batch</h3>
<div>This is the ideal solution, as it efficiently saves a list of records. This is what I will do when dealing with batch inserts.</div>
<div>
<br />
</div>
<div>
<b>BaseEntity</b>
</div>
<div>
<pre class="brush: java">@Getter
@Setter
@SuperBuilder
@NoArgsConstructor
@AllArgsConstructor
@MappedSuperclass
@JsonInclude(JsonInclude.Include.NON_NULL)
public abstract class BaseEntity implements Serializable {
@Serial
private static final long serialVersionUID = 3986494663579679129L;
@Id
@GeneratedValue(generator = "ID_GENERATOR", strategy = GenerationType.AUTO)
@Column(name = "id")
private Long id;
@Version
@Column(name = "version")
private Integer version;
}
</pre>
</div>
<div>
<ul style="text-align: left;">
<li>Line 14 - we are using an id generator that we will define in the promotion entity</li>
<li>Line 20 - we added a version field annotated with @Version for optimistic locking</li>
</ul>
</div>
<div>
<b>PromotionEntity</b>
</div>
<div>
<pre class="brush: java">@Getter
@Setter
@SuperBuilder
@NoArgsConstructor
@AllArgsConstructor
@Entity
@Table(name = "promotion")
@GenericGenerator(name = "ID_GENERATOR",
strategy = "org.hibernate.id.enhanced.SequenceStyleGenerator",
parameters = {@Parameter(name = "sequence_name", value = "promotion_seq")}
)
public class PromotionEntity extends BaseEntity {
@Column(name = "code")
private String code;
// ...
}
</pre>
</div>
<div>
<ul style="text-align: left;">
<li> Line 12 - we extend the BaseEntity</li>
<li>Line 8 - we create a generic sequence generator that will map to the ID_GENERATOR defined in the BaseEntity</li>
</ul>
<div>Let's check our revised service layer. I would use a CompletableFuture that I can chain if needed.</div>
</div>
<div>
<br />
</div>
<div style="text-align: left;">
<pre class="brush: java">@Transactional
public CompletableFuture<Void> createPromotions(Promotions promotions) {
log.debug("Creating promotions");
return CompletableFuture.runAsync(() -> promotionRepository.saveAll(
promotions.getValues().stream()
.map(service2EntityMapper::asPromotionEntity)
.toList())
);
}
</pre>
</div>
<div style="text-align: left;">
<ul>
<li>Line 1 - annotate the method with Transaction so that it can revert when an error occurred</li>
<li>Line 2 - our service method that returns a CompletableFuture<Void></li>
<li>Line 6 - here, we wrap our saveAll method inside an async block</li>
<li>Line 7 - we stream to the list of promotions</li>
<li>Line 8 - we convert each of the POJO to an entity</li>
<li>Line 9 - we collect the list of entities</li>
<li>Line 6 - we save all the entities, again it doesn't necessarily mean that we're saving in batch</li>
</ul>
<div>To enable batch saving, we must make changes to our configuration file.</div>
<br />
</div>
<div>
<b>application.xml</b>
</div>
<div style="text-align: left;">
<pre class="brush: java">spring:
application:
name: czetsuyatech.com
datasource:
url: jdbc:h2:mem:testdb
driverClassName: org.h2.Driver
username: sa
password:
jpa:
database-platform: org.hibernate.dialect.H2Dialect
properties:
hibernate:
jdbc:
batch_size: 5
generate_statistics: true
show-sql: false
flyway:
enabled: true
locations: classpath:db/migrations
baseline-on-migrate: true
user: sa
password: sa
h2:
console:
enabled: true
path: /db
</pre>
</div>
<div style="text-align: left;"><ul><li>Line 4 - our h2 data source</li><li>
Line 14 - you need to define the batch size. Along with sequenced id, it enables batch saving</li><li>Line 15 - useful when checking that you have successfully enabled batch</li><li>Line 17 - is all about the flyway configuration</li></ul></div>
<h4 style="text-align: left;"> Summary </h4>
<div>
<ul>
<li>It's always a good idea to wrap the create/update/delete operations in a transaction</li>
<li>Use stream when dealing with lists of entities</li>
<li>Use DTOs and POJOs when passing data from one layer to another</li><li>Use a custom sequence generator</li><li>Define batch_size hibernate property in application XML</li><li>I would use this approach to speed up the processing of records. The downside is that each batch is managed in one transaction, which means failing one record in that batch means the whole set will not be saved.</li>
</ul><div><h2>3. Challenge</h2></div><div>Using the above approach, try to save records in different sizes: 100, 1000, 10000, and 1000000. You should be able to see a noticeable improvement in inserts.</div><div><br /></div><h2 style="text-align: left;">4. GitHub Repository</h2><div><ul style="text-align: left;"><li><a href="https://github.com/czetsuya/test-powerledger">https://github.com/czetsuya/test-powerledger</a></li></ul></div>
</div>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-28207047341953752902022-11-26T13:51:00.004+08:002022-11-26T13:51:48.316+08:00How to Create a Bean of Type FeignClient<p>This class creates a bean that returns FeignClient constructed using FeignHttpClientProperties properties.</p>
<pre class="brush: java">package com.czetsuyatech.traceability.feign.config;
import com.amazonaws.xray.proxies.apache.http.HttpClientBuilder;
import feign.Client;
import feign.httpclient.ApacheHttpClient;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.springframework.cloud.openfeign.FeignFormatterRegistrar;
import org.springframework.cloud.openfeign.support.FeignHttpClientProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.format.datetime.standard.DateTimeFormatterRegistrar;
@Configuration(proxyBeanMethods = false)
@ConditionalFeignEnabled
public class FeignClientConfiguration {
@Bean
public Client client(HttpClientBuilder httpClientBuilder, FeignHttpClientProperties httpClientProperties) {
httpClientBuilder = httpClientBuilder != null ? httpClientBuilder : HttpClientBuilder.create();
final PoolingHttpClientConnectionManager poolingHttpClientConnectionManager =
new PoolingHttpClientConnectionManager();
poolingHttpClientConnectionManager.setDefaultMaxPerRoute(100);
poolingHttpClientConnectionManager.setMaxTotal(100);
final RequestConfig defaultRequestConfig = RequestConfig.custom()
.setConnectionRequestTimeout(100)
.setConnectTimeout(httpClientProperties.getConnectionTimeout())
.setSocketTimeout(100)
.setRedirectsEnabled(httpClientProperties.isFollowRedirects())
.build();
return new ApacheHttpClient(
httpClientBuilder
.setConnectionManager(poolingHttpClientConnectionManager)
.setDefaultRequestConfig(defaultRequestConfig)
.build()
);
}
@Bean
public FeignFormatterRegistrar localDateFeignFormatterRegister() {
return registry -> {
DateTimeFormatterRegistrar registrar = new DateTimeFormatterRegistrar();
registrar.setUseIsoFormat(true);
registrar.registerFormatters(registry);
};
}
}
</pre>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-70608782624326056192022-11-24T17:42:00.004+08:002022-11-24T17:46:49.537+08:00How to Manage Dates and Timezone in Code and Data<h2 style="text-align: left;">1. Introduction</h2><p>In a software application, it's almost certain that we have to deal with the date in one way or another. And it's always a problem when working on a multi-timezone application.</p><p>It's always important that there is consistency and smooth conversion of date information across all applications and services.</p><h2 style="text-align: left;">2. Rules for Managing Date Information</h2><p>Here are some rules that you can apply to manage the date.</p><p>2.1 The date information must be stored in UTC format in the database.</p><p>2.2 Date should be managed in UTC format in code.</p><p>2.3 Backend services must return a UTC format of the date.</p><p>2.4 Frontend must convert the date from UTC to the client's local timezone.</p><p>2.5 Frontend must use a standard format of the date. Preferably ISO 8601.</p><p>2.6 When the front end sends a date to the backend, it must be in ISO 8601 format. This will make conversion easier in the backend back to UTC format.</p><p>2.7 Take advantage of available libraries that can do date manipulation. </p><h2 style="text-align: left;">3. Available Date Libraries</h2><h3 style="text-align: left;">3.1 Java</h3><div><ul style="text-align: left;"><li>Joda-Time (<a href="https://www.joda.org/joda-time">https://www.joda.org/joda-time</a>)</li><li>Java-8 Date/Time</li><li>Commons Lang (<a href="https://commons.apache.org/proper/commons-lang">https://commons.apache.org/proper/commons-lang</a>)</li></ul><h3 style="text-align: left;">3.2 Javascript</h3></div><div><ul style="text-align: left;"><li>Moment (<a href="https://momentjs.com">https://momentjs.com</a>)</li></ul></div><div><br /></div>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0tag:blogger.com,1999:blog-6581802047354010081.post-20188306532200642672022-11-02T11:32:00.016+08:002022-11-20T11:15:59.603+08:00How to Configure Multitenancy in Keycloak with Spring API Gateway Integration<h2 style="text-align: left;">1. Introduction</h2>
<div>This article aims to demonstrate how multi-tenancy can be achieved in a microservice using Keycloak and Spring with Spring API Gateway in front of the services.</div>
<div>
<br />
</div>
<div>Spring API Gateway brings an out-of-the-box flexible routing with security, resilience, and monitoring management. How does Keycloak validate realm information and forward it to microservices?</div>
<div>
<br />
</div>
<div>In the example microservice project, the tenant configuration will be loaded based on the Keycloak-Realm header information to authenticate and authorize a user correctly.</div>
<h2 style="text-align: left;">2. The Problem</h2>
<div>In every software application, there is a need to create and manage users. We will design a software architecture where the client request is handled by an API Gateway and routed to the user service.</div><div><br /></div><h3 style="text-align: left;">2.1 Multi-Tenancy</h3><div>It is where you have a single software application that serves multiple customers or clients without sharing their confidential information. For example, multiple entities, such as telecom or electricity companies, can use a billing platform. Each entity will have access to the application without being able to look at the data of the other tenants.</div>
<div>
<br />
</div>
<div>Use Case Diagram.</div>
<div>
<br />
</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/a/AVvXsEgULYVrH-y31wBZBwsuuRJNexF24jl_yL6BuhopJJ-OP3q_t1dcPSr-ofxSriOX2tbN0GpPNnmy9M7RmOR72u5iUqNQEGwErmCjkKHU3335IC63jyfL3wxKSoNoD0ckSHviI86X0obB_oqeOVHoRTRv6xXpb8O37M--7BN6c3mvEjOzXozAVMJnQ3VYFQ" style="margin-left: 1em; margin-right: 1em;">
<img data-original-height="392" data-original-width="362" src="https://blogger.googleusercontent.com/img/a/AVvXsEgULYVrH-y31wBZBwsuuRJNexF24jl_yL6BuhopJJ-OP3q_t1dcPSr-ofxSriOX2tbN0GpPNnmy9M7RmOR72u5iUqNQEGwErmCjkKHU3335IC63jyfL3wxKSoNoD0ckSHviI86X0obB_oqeOVHoRTRv6xXpb8O37M--7BN6c3mvEjOzXozAVMJnQ3VYFQ=s16000" />
</a>
</div>
<br />
</div>
<div>Sequence Diagram</div>
<div>
<br />
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1v5W9dl2szSRZ1HpABBcm3yGcBb52_ZdYso4n04p_nUTRHj5mCCSlmY2MlnGL3bWhfl6eDH5hZdLqKM7w4tfwrcxJ9rW_yf7Qh1cHdIpAX-Vbmeqsnw1-y-XqyxxvEuSwZV3l7QitAS_S5yCXICRIkzphUjoEfthpyLqaCnR92pscNaze3oWdA3LnoA/s701/multi-tenant-sequence-diagram.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="409" data-original-width="701" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1v5W9dl2szSRZ1HpABBcm3yGcBb52_ZdYso4n04p_nUTRHj5mCCSlmY2MlnGL3bWhfl6eDH5hZdLqKM7w4tfwrcxJ9rW_yf7Qh1cHdIpAX-Vbmeqsnw1-y-XqyxxvEuSwZV3l7QitAS_S5yCXICRIkzphUjoEfthpyLqaCnR92pscNaze3oWdA3LnoA/s16000/multi-tenant-sequence-diagram.png" />
</a>
</div>
<h2 style="text-align: left;">3. Keycloak</h2>
<div>Keycloak is an open-source identity and access management platform. It provides out-of-the-box features such as authentication and authorization. </div>
<div>
<br />
</div>
<div>For this exercise, we will use a custom Keycloak server from <a href="https://github.com/czetsuyatech/ct-keycloak-iam">https://github.com/czetsuyatech/ct-keycloak-iam</a>. This project downloads a specific version of the Keycloak server and builds customizations such as theme, provider, realm, and more. </div>
<div>
<br />
</div>
<div>This exercise needs two tenants to demonstrate multi-tenancy, and thus the need to create two Keycloak realms.</div>
<h2 style="text-align: left;">4. Microservice Project in Spring</h2>
<div>Create a Spring service project and name it user-services. It will have one controller, UserController, and one endpoint that returns the information of the current log user, depending on which tenant the user belongs to.</div>
<div>
<br />
</div>
<div>In the getUserInfo method below, we are getting the Keycloak token from the security context and getting the principal. We can then typecast the object into KeycloakPrincipal, where we can get the accessToken and the user's primary attributes, such as first and last name.</div>
<pre class="brush: java">@GetMapping("/user-info")
public String userInfo() {
return getUserInfo();
}
@SuppressWarnings("unchecked")
private String getUserInfo() {
KeycloakAuthenticationToken authentication = (KeycloakAuthenticationToken) SecurityContextHolder.getContext()
.getAuthentication();
final Principal principal = (Principal) authentication.getPrincipal();
String tokenInfo = null;
if (principal instanceof KeycloakPrincipal) {
KeycloakPrincipal<keycloaksecuritycontext> kPrincipal = (KeycloakPrincipal<keycloaksecuritycontext>) principal;
KeycloakSecurityContext ksc = kPrincipal.getKeycloakSecurityContext();
IDToken token = ksc.getIdToken();
AccessToken accessToken = kPrincipal.getKeycloakSecurityContext().getToken();
tokenInfo = accessToken.getSubject();
// this value is the one use to call another service as bearer token
// Authorization : Bearer kcs.getTokenString()
// use this link to read the token https://jwt.io
return String.format("Hello %s %s [subject=%s]", accessToken.getGivenName(), accessToken.getFamilyName(),
tokenInfo);
}
return "Hello World";
}
</pre>
<h2 style="text-align: left;">5. Spring API Gateway Project</h2>
<div>We need another Spring project that will have a dependency on the spring-cloud-starter-gateway. In this project, we must define request routing and Jwt issuer URIs for each realm.</div>
<div>
<br />
</div>
<div>With TokenRelayGatewayFilterFactory, it acts as an OAuth2 consumer and forwards the incoming token to the outgoing request resource, in this case to the user service.</div>
<pre class="brush: java">@Bean
public RouteLocator gatewayRouter(RouteLocatorBuilder builder, TokenRelayGatewayFilterFactory filterFactory) {
return builder.routes()
.route(p -> p.path("/test")
.uri("http://httpbin.org"))
.route("users", p -> p.path("/users/**")
.filters(f -> f.filter(filterFactory.apply()))
.uri("http://localhost:8001"))
.build();
}
</pre>
<h2 style="text-align: left;">6. Testing</h2>
<h3 style="text-align: left;">6.1 Keycloak</h3>
<div>You can use the custom Keycloak project I linked in the references section. You will need two realms to test the application. Included in the project.</div>
<div>
<br />
</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOnbyCbNt4esk-KZdF2LZ_5t_dRP9lc2_l11w1Crq7CpvQZkzhXHBA94kI4FgEkLrVTRj71mwrUtQVwZJKGzXQVzjY7ilU9NIa-QGtiOPhOuGU8bygjqKN_f9w5dGYDquPh-5mY6ofWXV2Kh5HLqYBC9k6Zi81PcaoyY7u7mGM4hSDz_BfWArhXSIDDg/s2394/keycloak_logs.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="728" data-original-width="2394" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOnbyCbNt4esk-KZdF2LZ_5t_dRP9lc2_l11w1Crq7CpvQZkzhXHBA94kI4FgEkLrVTRj71mwrUtQVwZJKGzXQVzjY7ilU9NIa-QGtiOPhOuGU8bygjqKN_f9w5dGYDquPh-5mY6ofWXV2Kh5HLqYBC9k6Zi81PcaoyY7u7mGM4hSDz_BfWArhXSIDDg/s16000/keycloak_logs.png" />
</a>
</div>
</div>
<h3 style="text-align: left;">6.2 Postman Collection</h3>
<div>I am attaching a Postman collection that you can use to test the endpoints. Included in the project.</div>
<p>Tenant1 / User1</p>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCILriiQRlC1UvN3LkQPwG2v5DOWdx_czWC1h-lH2TIvUz7cghQPmNFRDTr-p_VP1nDsNuPNoX3u9CI_XFX-gzdZ3dakjAdDJqKxgrp6tOenqRwIPrET2Ym0I4k1_W1yJnqT1I8HR1nU40qxcKGlIW_2bE5pNj2Cr7IOnvZrfQiz19YwE8hGYX5hezLg/s2411/postman-logs.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="1089" data-original-width="2411" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCILriiQRlC1UvN3LkQPwG2v5DOWdx_czWC1h-lH2TIvUz7cghQPmNFRDTr-p_VP1nDsNuPNoX3u9CI_XFX-gzdZ3dakjAdDJqKxgrp6tOenqRwIPrET2Ym0I4k1_W1yJnqT1I8HR1nU40qxcKGlIW_2bE5pNj2Cr7IOnvZrfQiz19YwE8hGYX5hezLg/s16000/postman-logs.png" />
</a>
</div>
<p>Tenant2 / User2</p>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDxFgdaE6VtHxnfD5rCt22yItl-WRPo2xhGMLWSMDx_uwUgtYuf1p8EOIArhsS4wBK6JXnVvYw2BF9aiWXWKUTzyOyQ5OUNNRiVhJMPrHHAGYnKiPB78-8qgde4GxD1tEbBOO0pdrQ4CjcTCqgw5qttMiNYoOjMBAM-BA3N71A_oVUaNChDrgTF60hNw/s2411/postman-logs2.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="1164" data-original-width="2411" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDxFgdaE6VtHxnfD5rCt22yItl-WRPo2xhGMLWSMDx_uwUgtYuf1p8EOIArhsS4wBK6JXnVvYw2BF9aiWXWKUTzyOyQ5OUNNRiVhJMPrHHAGYnKiPB78-8qgde4GxD1tEbBOO0pdrQ4CjcTCqgw5qttMiNYoOjMBAM-BA3N71A_oVUaNChDrgTF60hNw/s16000/postman-logs2.png" />
</a>
</div>
<h3>6.3 User Service</h3>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhebGh4qCgYbi9naoLQjVEtw90_OOc69Blkgw98sq55epGu_c6Y-JMxF7N0_T0zHPjuKUZEUooTn0inaAqOXM5xdAeSE2QzjxAOsSTcukkO4NcHvCHSIy5GnFd2l3B8A9Mv_z7UhlXBxwOvrhfwIn3Qs8W6QiBmBq4-rcxKUECsqQxxthuLQspWXf0DXg/s2919/user-services-log.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="132" data-original-width="2919" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhebGh4qCgYbi9naoLQjVEtw90_OOc69Blkgw98sq55epGu_c6Y-JMxF7N0_T0zHPjuKUZEUooTn0inaAqOXM5xdAeSE2QzjxAOsSTcukkO4NcHvCHSIy5GnFd2l3B8A9Mv_z7UhlXBxwOvrhfwIn3Qs8W6QiBmBq4-rcxKUECsqQxxthuLQspWXf0DXg/s16000/user-services-log.png" />
</a>
</div>
<h3 style="text-align: left;">6.4 API Gateway</h3>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgS8Eoqn_Oz5GYKRDQivDySZF1TcaKWLfcMkPK2BKRVyA1xcc1pdAiPoBnaVx7TpevTd_IYoJ0Ow0P_QtFP9cq8-l35IGjjhUpCmI547Mg0-UIH6F90FgpFIoxDwnrsc7JMXjqsoosyKLnccwvrE1GZq9t9yZRYyABvOqxlLsxQoGE7f9vc3MBhPwv9Qw/s2919/api-gateway-logs.png" style="margin-left: 1em; margin-right: 1em;">
<img border="0" data-original-height="89" data-original-width="2919" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgS8Eoqn_Oz5GYKRDQivDySZF1TcaKWLfcMkPK2BKRVyA1xcc1pdAiPoBnaVx7TpevTd_IYoJ0Ow0P_QtFP9cq8-l35IGjjhUpCmI547Mg0-UIH6F90FgpFIoxDwnrsc7JMXjqsoosyKLnccwvrE1GZq9t9yZRYyABvOqxlLsxQoGE7f9vc3MBhPwv9Qw/s16000/api-gateway-logs.png" />
</a>
</div>
<h2 style="text-align: left;">7. Improvements</h2>
<div>
<ul style="text-align: left;">
<li>To avoid redeploying a war file, add a new tenant keycloak.json file and external storage such as S3.</li>
</ul>
<h2 style="text-align: left;">8. References</h2>
</div>
<div>
<ul style="text-align: left;">
<li>
<a href="https://github.com/czetsuyatech/ct-keycloak-iam">https://github.com/czetsuyatech/ct-multi-tenant-keycloak-spring</a> (sponsors only)
</li>
<li>
<a href="https://github.com/czetsuyatech/ct-keycloak-iam">https://github.com/czetsuyatech/ct-keycloak-iam</a>
</li>
<li>
<a href="https://github.com/sponsors/czetsuya">https://github.com/sponsors/czetsuya</a>
</li>
</ul>
</div>
<h2 style="text-align: left;">9. Youtube Demo</h2>
<div class="separator" style="clear: both; text-align: center;"><iframe allowfullscreen="" class="BLOG_video_class" height="360" src="https://www.youtube.com/embed/qaKvEs7oAf4" width="640" youtube-src-id="qaKvEs7oAf4"></iframe></div>czetsuyahttp://www.blogger.com/profile/05992265396469184599noreply@blogger.com0