[Ru_ngi] RDIG CRL problem

Thu May 16 14:53:32 MSK 2024

Добрый день.
В ИЯИ в районе 8-9 мая возникла другая проблема. Перестали отрабатывать 
пилоты cms на arc ce, выставили тикет 166770
Все sgmcms тесты OK. Я запускала тесты с lxplus.cern.ch и получила все 
результаты и по
https://grinr04.inr.troitsk.ru:443/arex/rest/1.0/jobs/
и по
gsiftp://grinr05.inr.troitsk.ru:2811/jobs/

B ws-interface.log

[2024-05-16 05:23:50] [Arc.MCC.HTTP] [DEBUG] [28256/814] < POST 
/arex/rest/1.0/jobs?action=new HTTP/1.1
[2024-05-16 05:23:50] [Arc.MCC.HTTP] [DEBUG] [28256/814] < Host: 
grinr05.inr.troitsk.ru
[2024-05-16 05:23:50] [Arc.MCC.HTTP] [DEBUG] [28256/814] < Content-Type: 
application/xml
[2024-05-16 05:23:50] [Arc.MCC.HTTP] [DEBUG] [28256/814] < Accept: 
application/json
[2024-05-16 05:23:50] [Arc.MCC.HTTP] [DEBUG] [28256/814] < 
Content-Length: 5814
[2024-05-16 05:23:50] [Arc.MCC.HTTP] [DEBUG] [28256/814] < Expect: 
100-continue
[2024-05-16 05:23:50] [Arc.SecHandler] [DEBUG] [28256/814] OTokens: 
Handle
[2024-05-16 05:23:50] [Arc.SecHandler] [DEBUG] [28256/814] OTokens: 
Handle: message
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
message
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
ENDPOINT = http://2a0e:e140::135:443/arex/rest/1.0/jobs?action=new
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
HTTP:ENDPOINT = /arex/rest/1.0/jobs?action=new
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
HTTP:METHOD = POST
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
HTTP:accept = application/json
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
HTTP:content-length = 5814
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
HTTP:content-type = application/xml
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
HTTP:expect = 100-continue
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
HTTP:host = grinr05.inr.troitsk.ru
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
TCP:ENDPOINT = ://2a0e:e140::135:443
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
TCP:HOST = 2a0e:e140::135
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
TCP:PORT = 443
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
TCP:REMOTEHOST = 2001:1458:d00:3a::100:110
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
TCP:REMOTEPORT = 1908
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
TLS:CADN = /C=ch/O=CERN/CN=CERN Root Certification Authority 2
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
TLS:IDENTITYDN = 
/DC=ch/DC=cern/OU=computers/CN=cmspilot02/vocms080.cern.ch
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
TLS:LOCALDN = /C=RU/O=RDIG/OU=hosts/OU=inr.ru/CN=grinr05.inr.troitsk.ru
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
TLS:PEERDN = 
/DC=ch/DC=cern/OU=computers/CN=cmspilot02/vocms080.cern.ch/CN=1744970950
[2024-05-16 05:23:50] [Arc.OTokensSH] [DEBUG] [28256/814] OTokens: Attr: 
token: (empty)
[2024-05-16 05:23:50] [Arc.SecHandler] [ERROR] [28256/814] Failed to 
create OTokens security attributes
[2024-05-16 05:23:50] [Arc.AuthUserVOMS] [VERBOSE] [28256/814] Rule: vo: 
cms
[2024-05-16 05:23:50] [Arc.AuthUserVOMS] [VERBOSE] [28256/814] Rule: 
group: *
[2024-05-16 05:23:50] [Arc.AuthUserVOMS] [VERBOSE] [28256/814] Rule: 
role: pilot
[2024-05-16 05:23:50] [Arc.AuthUserVOMS] [VERBOSE] [28256/814] Rule: 
capabilities: *

В arex-jobs.log
2024-05-16 05:23:50 Started - job id: 
xrdKDmmxfR5nOw7A1pXY3zgoABFKDmABFKDmuWMKDmNCFKDmsCHbfm, unix user: 
55301:1402, name: "", owner: 
"/DC=ch/DC=cern/OU=computers/CN=cmspilot02/vocms080.cern.ch", lrms: pbs, 
queue: cms

2024-05-16 05:39:10 Finished - job id: 
xrdKDmmxfR5nOw7A1pXY3zgoABFKDmABFKDmuWMKDmNCFKDmsCHbfm, unix user: 
55301:1402, name: "", owner: 
"/DC=ch/DC=cern/OU=computers/CN=cmspilot02/vocms080.cern.ch", lrms: pbs, 
queue: cms, failure: "User file: 
/credential_CMSG-v1_0.main-arc_CMSHTPC_T2_RU_INR_grinr05.idtoken - 
Timeout waiting.User file: /tokens.tgz - Timeout waiting.User file: 
/glidein_startup.sh - Timeout waiting."

В blahp.log-20240516
grep xrdKDmmxfR5nOw7A1pXY3zgoABFKDmABFKDmuWMKDmNCFKDmsCHbfm 
blahp.log-20240516
"timestamp=2024-05-16 02:23:50" 
"userDN=/DC=ch/DC=cern/OU=computers/CN=cmspilot02/vocms080.cern.ch" 
"userFQAN=/cms/Role=pilot" "userFQAN=/cms/Role=NULL" 
"userFQAN=/cms/dcms/Role=NULL" "userFQAN=/cms/escms/Role=NULL" 
"userFQAN=/cms/itcms/Role=NULL" "userFQAN=/cms/local/Role=NULL" 
"userFQAN=/cms/uscms/Role=NULL" 
"ceID=grinr05.inr.troitsk.ru:2811/nordugrid-torque-cms" 
"jobID=xrdKDmmxfR5nOw7A1pXY3zgoABFKDmABFKDmuWMKDmNCFKDmsCHbfm" "lrmsID=" 
"localUser=55301" 
"clientID=https://grinr05.inr.troitsk.ru:443/arex/xrdKDmmxfR5nOw7A1pXY3zgoABFKDmABFKDmuWMKDmNCFKDmsCHbfm"

Смотрела tcpdump, пакеты с
  IP6 2a0e:e140::135.443 > 2001:1458:d00:3a::100:110.1908
и обратно
  IP6 2001:1458:d00:3a::100:110.1908 > 2a0e:e140::135.443
бегают
С уважением,
    Людмила.

Viktor Kotliar писал 2024-05-16 11:40:
> Всем привет!
> Нам тут выставили тикет [1]
> 
> При работе с нашим SE некоторые получают "No CRLs found for issuer
> "cn=Russian Data-Intensive Grid CA,o=RDIG,c=RU", это проблема не
> нашего сайта, я верно понимаю? Вопрос только это проблема их клиента
> или RDIG CA. Никто больше не словил такого?
> 
> Вот лог с ошибкой [2]  Либо кусочек из лога [3]
> 
> С уважением
> Виктор Котляр
> 
> 
> [1]
> ```
> https://ggus.eu/index.php?mode=ticket_info&ticket_id=166785
> ```
> 
> [2]
> ```
> https://fts-atlas-008.cern.ch:8449/var/log/fts3/transfers/2024-05-14/se0002.m45.ihep.su__clrlcgse01.in2p3.fr/2024-05-14-2359__se0002.m45.ihep.su__clrlcgse01.in2p3.fr__6500003593__f57be15c-124d-11ef-9b33-fa163ea7ee69
> ```
> 
> [3]
> ```
> NFO    Wed, 15 May 2024 01:59:19 +0200; Davix: Hop:
> https://clrlcgse01.in2p3.fr:443/dpm/in2p3.fr/home/atlas/atlasdatadisk/rucio/data17_13TeV/a2/e2/DAOD_PHYS.37020486._001025.pool.root.1
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix:
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: > COPY
> /dpm/in2p3.fr/home/atlas/atlasdatadisk/rucio/data17_13TeV/a2/e2/DAOD_PHYS.37020486._001025.pool.root.1
> HTTP/1.1
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: > Host: 
> clrlcgse01.in2p3.fr
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: > Accept: */*
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: > Source:
> https://se0002.m45.ihep.su:2880/atlas/atlasdatadisk/rucio/data17_13TeV/a2/e2/DAOD_PHYS.37020486._001025.pool.root.1?<redacted>
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: > X-Number-Of-Streams: 
> 1
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: > Secure-Redirection: 1
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: > ClientInfo:
> job-id=f57be15c-124d-11ef-9b33-fa163ea7ee69;file-id=6500003593;retry=0
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: > TransferMetadata:
> eyJrZXkiOiAibXkgbWV0YWRhdGEifQ==
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: >
> TransferHeaderAuthorization:
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: > Credential: none
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: > Authorization:
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: >
> RequireChecksumVerification: false
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: > TransferMetadata:
> eyJjb2xsb2NhdGlvbl9oaW50cyI6IHt9fQ==
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: > SciTag: 145
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: > User-Agent:
> libdavix/0.8.6.1.eddf9a5 libcurl/7.76.1
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix:
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: < HTTP/1.1 202 Accepted
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: < Date: Tue, 14 May
> 2024 23:59:19 GMT
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: < Server: dCache/8.2.40
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: < Content-Type:
> text/perf-marker-stream
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: < Transfer-Encoding: 
> chunked
> INFO    Wed, 15 May 2024 01:59:19 +0200; Davix: PerformanceMarker:
> failure: The peer's certificate with subject's DN
> CN=se0002.m45.ihep.su,OU=ihep.su,OU=hosts,O=RDIG,C=RU was rejected.
> The peer's certificate status is: FAILED The following validation
> errors were found:;error at position 0 in chain, problematic
> certificate subject:
> CN=se0002.m45.ihep.su,OU=ihep.su,OU=hosts,O=RDIG,C=RU (category: CRL):
> No valid CRL was found for the CA which issued the chain Cause: No
> CRLs found for issuer "cn=Russian Data-Intensive Grid CA,o=RDIG,c=RU"
> 
> INFO    Wed, 15 May 2024 01:59:19 +0200; Gfal2: Copy failed with mode
> 3rd pull: Transfer failure: The peer's certificate with subject's DN
> CN=se0002.m45.ihep.su,OU=ihep.su,OU=hosts,O=RDIG,C=RU was rejected.
> The peer's certificate status is: FAILED The following validation
> errors were found:;error at position 0 in chain, problematic
> certificate subject:
> CN=se0002.m45.ihep.su,OU=ihep.su,OU=hosts,O=RDIG,C=RU (category: CRL):
> No valid CRL was found for the CA which issued the chain Cause: No
> CRLs found for issuer "cn=Russian Data-Intensive Grid CA,o=RDIG,c=RU"
> ```