Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2094 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2485. Min Act: -0.0000

Data Index: 1389791 (Python Code)

Max Activating Token Index: 694

Text #1

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2477. Min Act: -0.0000

Data Index: 1092778 (C4 (Web Text))

Max Activating Token Index: 404

Text #2

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2553. Min Act: -0.0000

Data Index: 1374089 (Python Code)

Max Activating Token Index: 748

Text #3

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2580. Min Act: -0.0000

Data Index: 1374700 (Python Code)

Max Activating Token Index: 90

Text #4

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2472. Min Act: -0.0000

Data Index: 1465257 (Python Code)

Max Activating Token Index: 74

Text #5

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2562. Min Act: -0.0000

Data Index: 197705 (C4 (Web Text))

Max Activating Token Index: 302

Text #6

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2344. Min Act: -0.0000

Data Index: 392990 (C4 (Web Text))

Max Activating Token Index: 943

Text #7

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2495. Min Act: -0.0000

Data Index: 490023 (C4 (Web Text))

Max Activating Token Index: 503

Text #8

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2401. Min Act: -0.0000

Data Index: 1447369 (Python Code)

Max Activating Token Index: 16

Text #9

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2295. Min Act: -0.0000

Data Index: 1651149 (Python Code)

Max Activating Token Index: 377

Text #10

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2200. Min Act: -0.0000

Data Index: 1509545 (Python Code)

Max Activating Token Index: 844

Text #11

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2283. Min Act: -0.0000

Data Index: 1444200 (Python Code)

Max Activating Token Index: 119

Text #12

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2298. Min Act: -0.0000

Data Index: 1612600 (Python Code)

Max Activating Token Index: 100

Text #13

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2364. Min Act: -0.0000

Data Index: 1165210 (C4 (Web Text))

Max Activating Token Index: 14

Text #14

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2272. Min Act: -0.0000

Data Index: 1649089 (Python Code)

Max Activating Token Index: 772

Text #15

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2277. Min Act: -0.0000

Data Index: 170130 (C4 (Web Text))

Max Activating Token Index: 465

Text #16

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2346. Min Act: -0.0000

Data Index: 1127898 (C4 (Web Text))

Max Activating Token Index: 397

Text #17

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2227. Min Act: -0.0000

Data Index: 53866 (C4 (Web Text))

Max Activating Token Index: 564

Text #18

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2388. Min Act: -0.0000

Data Index: 801371 (C4 (Web Text))

Max Activating Token Index: 120

Text #19

Max Range: 0.2580. Min Range: -0.2580

Max Act: 0.2395. Min Act: -0.0000

Data Index: 1449797 (Python Code)

Max Activating Token Index: 495