Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1007 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.4126. Min Act: -0.0000

Data Index: 959626 (C4 (Web Text))

Max Activating Token Index: 689

Text #1

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.3795. Min Act: -0.0001

Data Index: 847138 (C4 (Web Text))

Max Activating Token Index: 169

Text #2

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.3804. Min Act: -0.0001

Data Index: 593611 (C4 (Web Text))

Max Activating Token Index: 19

Text #3

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.3713. Min Act: -0.0001

Data Index: 581964 (C4 (Web Text))

Max Activating Token Index: 183

Text #4

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.3879. Min Act: -0.0001

Data Index: 380361 (C4 (Web Text))

Max Activating Token Index: 897

Text #5

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.3628. Min Act: -0.0000

Data Index: 155499 (C4 (Web Text))

Max Activating Token Index: 703

Text #6

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.3436. Min Act: -0.0000

Data Index: 1251708 (C4 (Web Text))

Max Activating Token Index: 1008

Text #7

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.3334. Min Act: -0.0001

Data Index: 127216 (C4 (Web Text))

Max Activating Token Index: 705

Text #8

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.3367. Min Act: -0.0001

Data Index: 837214 (C4 (Web Text))

Max Activating Token Index: 438

Text #9

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.3474. Min Act: -0.0000

Data Index: 7911 (C4 (Web Text))

Max Activating Token Index: 701

Text #10

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.3307. Min Act: -0.0000

Data Index: 2232 (C4 (Web Text))

Max Activating Token Index: 257

Text #11

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.3106. Min Act: -0.0000

Data Index: 346343 (C4 (Web Text))

Max Activating Token Index: 428

Text #12

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.3203. Min Act: -0.0001

Data Index: 1272077 (C4 (Web Text))

Max Activating Token Index: 373

Text #13

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.2967. Min Act: -0.0000

Data Index: 38825 (C4 (Web Text))

Max Activating Token Index: 616

Text #14

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.2990. Min Act: -0.0001

Data Index: 735217 (C4 (Web Text))

Max Activating Token Index: 189

Text #15

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.2852. Min Act: -0.0001

Data Index: 1299967 (C4 (Web Text))

Max Activating Token Index: 673

Text #16

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.2775. Min Act: -0.0000

Data Index: 376053 (C4 (Web Text))

Max Activating Token Index: 300

Text #17

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.2760. Min Act: -0.0000

Data Index: 887696 (C4 (Web Text))

Max Activating Token Index: 588

Text #18

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.2817. Min Act: -0.0001

Data Index: 770712 (C4 (Web Text))

Max Activating Token Index: 532

Text #19

Max Range: 0.4126. Min Range: -0.4126

Max Act: 0.2737. Min Act: -0.0001

Data Index: 605611 (C4 (Web Text))

Max Activating Token Index: 678