Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4712 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.7613. Min Act: -0.0000

Data Index: 490134 (C4 (Web Text))

Max Activating Token Index: 656

Text #1

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.5264. Min Act: -0.0000

Data Index: 819394 (C4 (Web Text))

Max Activating Token Index: 469

Text #2

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.5360. Min Act: -0.0000

Data Index: 685380 (C4 (Web Text))

Max Activating Token Index: 469

Text #3

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.4328. Min Act: -0.0000

Data Index: 377130 (C4 (Web Text))

Max Activating Token Index: 188

Text #4

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.4265. Min Act: 0.0000

Data Index: 1141674 (C4 (Web Text))

Max Activating Token Index: 844

Text #5

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.5001. Min Act: -0.0000

Data Index: 344158 (C4 (Web Text))

Max Activating Token Index: 764

Text #6

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.4997. Min Act: -0.0000

Data Index: 967058 (C4 (Web Text))

Max Activating Token Index: 919

Text #7

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.4202. Min Act: -0.0000

Data Index: 801694 (C4 (Web Text))

Max Activating Token Index: 57

Text #8

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.4703. Min Act: -0.0000

Data Index: 278617 (C4 (Web Text))

Max Activating Token Index: 856

Text #9

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.2813. Min Act: -0.0000

Data Index: 573344 (C4 (Web Text))

Max Activating Token Index: 458

Text #10

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.4115. Min Act: -0.0000

Data Index: 528210 (C4 (Web Text))

Max Activating Token Index: 521

Text #11

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.2532. Min Act: -0.0000

Data Index: 831459 (C4 (Web Text))

Max Activating Token Index: 542

Text #12

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.2155. Min Act: -0.0000

Data Index: 328923 (C4 (Web Text))

Max Activating Token Index: 927

Text #13

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.2866. Min Act: -0.0000

Data Index: 816327 (C4 (Web Text))

Max Activating Token Index: 953

Text #14

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.3733. Min Act: -0.0000

Data Index: 348375 (C4 (Web Text))

Max Activating Token Index: 487

Text #15

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.3717. Min Act: -0.0000

Data Index: 280151 (C4 (Web Text))

Max Activating Token Index: 621

Text #16

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.2493. Min Act: -0.0000

Data Index: 79217 (C4 (Web Text))

Max Activating Token Index: 582

Text #17

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.2751. Min Act: -0.0000

Data Index: 496670 (C4 (Web Text))

Max Activating Token Index: 683

Text #18

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.1769. Min Act: -0.0000

Data Index: 335764 (C4 (Web Text))

Max Activating Token Index: 777

Text #19

Max Range: 1.7613. Min Range: -1.7613

Max Act: 1.2950. Min Act: -0.0000

Data Index: 576157 (C4 (Web Text))

Max Activating Token Index: 408