Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5003 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4886. Min Act: -0.0000

Data Index: 877468 (C4 (Web Text))

Max Activating Token Index: 635

Text #1

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4749. Min Act: -0.0000

Data Index: 1117866 (C4 (Web Text))

Max Activating Token Index: 931

Text #2

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4730. Min Act: -0.0000

Data Index: 824104 (C4 (Web Text))

Max Activating Token Index: 895

Text #3

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4666. Min Act: -0.0000

Data Index: 400186 (C4 (Web Text))

Max Activating Token Index: 569

Text #4

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4444. Min Act: -0.0000

Data Index: 542179 (C4 (Web Text))

Max Activating Token Index: 965

Text #5

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.5011. Min Act: -0.0000

Data Index: 628142 (C4 (Web Text))

Max Activating Token Index: 1011

Text #6

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4797. Min Act: -0.0000

Data Index: 574185 (C4 (Web Text))

Max Activating Token Index: 834

Text #7

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4659. Min Act: -0.0000

Data Index: 25625 (C4 (Web Text))

Max Activating Token Index: 610

Text #8

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4411. Min Act: -0.0000

Data Index: 112220 (C4 (Web Text))

Max Activating Token Index: 548

Text #9

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4339. Min Act: -0.0000

Data Index: 901888 (C4 (Web Text))

Max Activating Token Index: 824

Text #10

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4324. Min Act: -0.0000

Data Index: 166405 (C4 (Web Text))

Max Activating Token Index: 397

Text #11

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4635. Min Act: -0.0000

Data Index: 678726 (C4 (Web Text))

Max Activating Token Index: 365

Text #12

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4658. Min Act: -0.0000

Data Index: 507130 (C4 (Web Text))

Max Activating Token Index: 622

Text #13

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4560. Min Act: -0.0000

Data Index: 999423 (C4 (Web Text))

Max Activating Token Index: 350

Text #14

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4370. Min Act: -0.0000

Data Index: 216426 (C4 (Web Text))

Max Activating Token Index: 512

Text #15

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4458. Min Act: -0.0000

Data Index: 965548 (C4 (Web Text))

Max Activating Token Index: 378

Text #16

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4428. Min Act: -0.0000

Data Index: 664627 (C4 (Web Text))

Max Activating Token Index: 137

Text #17

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4512. Min Act: -0.0000

Data Index: 973310 (C4 (Web Text))

Max Activating Token Index: 230

Text #18

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4112. Min Act: -0.0000

Data Index: 173019 (C4 (Web Text))

Max Activating Token Index: 249

Text #19

Max Range: 0.5011. Min Range: -0.5011

Max Act: 0.4198. Min Act: -0.0000

Data Index: 1076609 (C4 (Web Text))

Max Activating Token Index: 883