Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4104 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.5901. Min Act: -0.0000

Data Index: 1001614 (C4 (Web Text))

Max Activating Token Index: 209

Text #1

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.6640. Min Act: -0.0000

Data Index: 1153560 (C4 (Web Text))

Max Activating Token Index: 568

Text #2

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.6453. Min Act: -0.0000

Data Index: 1253637 (C4 (Web Text))

Max Activating Token Index: 323

Text #3

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.4872. Min Act: -0.0000

Data Index: 673592 (C4 (Web Text))

Max Activating Token Index: 257

Text #4

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.4785. Min Act: -0.0000

Data Index: 160408 (C4 (Web Text))

Max Activating Token Index: 332

Text #5

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.5728. Min Act: -0.0000

Data Index: 52949 (C4 (Web Text))

Max Activating Token Index: 199

Text #6

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.5582. Min Act: -0.0000

Data Index: 839989 (C4 (Web Text))

Max Activating Token Index: 695

Text #7

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.5395. Min Act: -0.0000

Data Index: 919799 (C4 (Web Text))

Max Activating Token Index: 703

Text #8

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.3878. Min Act: -0.0000

Data Index: 266572 (C4 (Web Text))

Max Activating Token Index: 717

Text #9

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.4821. Min Act: -0.0000

Data Index: 774368 (C4 (Web Text))

Max Activating Token Index: 384

Text #10

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.4890. Min Act: -0.0000

Data Index: 1148231 (C4 (Web Text))

Max Activating Token Index: 24

Text #11

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.4446. Min Act: -0.0000

Data Index: 1036364 (C4 (Web Text))

Max Activating Token Index: 113

Text #12

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.4443. Min Act: -0.0000

Data Index: 802936 (C4 (Web Text))

Max Activating Token Index: 334

Text #13

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.4389. Min Act: -0.0000

Data Index: 132489 (C4 (Web Text))

Max Activating Token Index: 47

Text #14

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.4248. Min Act: -0.0000

Data Index: 70729 (C4 (Web Text))

Max Activating Token Index: 841

Text #15

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.3915. Min Act: -0.0000

Data Index: 893397 (C4 (Web Text))

Max Activating Token Index: 528

Text #16

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.5305. Min Act: -0.0000

Data Index: 111049 (C4 (Web Text))

Max Activating Token Index: 258

Text #17

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.3845. Min Act: -0.0000

Data Index: 1178023 (C4 (Web Text))

Max Activating Token Index: 507

Text #18

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.2539. Min Act: -0.0000

Data Index: 410767 (C4 (Web Text))

Max Activating Token Index: 476

Text #19

Max Range: 2.6640. Min Range: -2.6640

Max Act: 2.2502. Min Act: -0.0000

Data Index: 160570 (C4 (Web Text))

Max Activating Token Index: 592