Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4082 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.7157. Min Act: -0.0000

Data Index: 1060748 (C4 (Web Text))

Max Activating Token Index: 458

Text #1

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.4389. Min Act: -0.0000

Data Index: 767321 (C4 (Web Text))

Max Activating Token Index: 650

Text #2

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.4489. Min Act: -0.0000

Data Index: 277392 (C4 (Web Text))

Max Activating Token Index: 273

Text #3

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.3275. Min Act: 0.0000

Data Index: 292417 (C4 (Web Text))

Max Activating Token Index: 902

Text #4

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.3443. Min Act: -0.0000

Data Index: 393963 (C4 (Web Text))

Max Activating Token Index: 215

Text #5

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.1132. Min Act: -0.0000

Data Index: 994446 (C4 (Web Text))

Max Activating Token Index: 303

Text #6

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.2593. Min Act: -0.0000

Data Index: 704492 (C4 (Web Text))

Max Activating Token Index: 449

Text #7

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.3667. Min Act: -0.0000

Data Index: 425908 (C4 (Web Text))

Max Activating Token Index: 751

Text #8

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.2914. Min Act: -0.0000

Data Index: 450896 (C4 (Web Text))

Max Activating Token Index: 480

Text #9

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.2520. Min Act: -0.0000

Data Index: 513281 (C4 (Web Text))

Max Activating Token Index: 527

Text #10

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.2327. Min Act: -0.0000

Data Index: 509857 (C4 (Web Text))

Max Activating Token Index: 646

Text #11

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.1390. Min Act: -0.0000

Data Index: 559089 (C4 (Web Text))

Max Activating Token Index: 525

Text #12

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.1719. Min Act: -0.0000

Data Index: 484057 (C4 (Web Text))

Max Activating Token Index: 243

Text #13

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.1643. Min Act: -0.0000

Data Index: 14018 (C4 (Web Text))

Max Activating Token Index: 198

Text #14

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.1915. Min Act: -0.0000

Data Index: 1045267 (C4 (Web Text))

Max Activating Token Index: 945

Text #15

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.1586. Min Act: 0.0000

Data Index: 1039265 (C4 (Web Text))

Max Activating Token Index: 633

Text #16

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.1522. Min Act: -0.0000

Data Index: 1251944 (C4 (Web Text))

Max Activating Token Index: 494

Text #17

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.1286. Min Act: -0.0000

Data Index: 845012 (C4 (Web Text))

Max Activating Token Index: 693

Text #18

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.1369. Min Act: -0.0000

Data Index: 845830 (C4 (Web Text))

Max Activating Token Index: 809

Text #19

Max Range: 1.7157. Min Range: -1.7157

Max Act: 1.1472. Min Act: -0.0000

Data Index: 192836 (C4 (Web Text))

Max Activating Token Index: 425