Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5242 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.2623. Min Range: -3.2623

Max Act: 3.2623. Min Act: -0.0000

Data Index: 915533 (C4 (Web Text))

Max Activating Token Index: 131

Text #1

Max Range: 3.2623. Min Range: -3.2623

Max Act: 3.1794. Min Act: -0.0000

Data Index: 1056031 (C4 (Web Text))

Max Activating Token Index: 937

Text #2

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.9831. Min Act: -0.0000

Data Index: 1240115 (C4 (Web Text))

Max Activating Token Index: 781

Text #3

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.8566. Min Act: -0.0000

Data Index: 1182690 (C4 (Web Text))

Max Activating Token Index: 431

Text #4

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.8423. Min Act: -0.0000

Data Index: 963138 (C4 (Web Text))

Max Activating Token Index: 796

Text #5

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.6684. Min Act: -0.0000

Data Index: 924720 (C4 (Web Text))

Max Activating Token Index: 266

Text #6

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.7857. Min Act: -0.0000

Data Index: 305141 (C4 (Web Text))

Max Activating Token Index: 873

Text #7

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.4564. Min Act: -0.0000

Data Index: 102917 (C4 (Web Text))

Max Activating Token Index: 697

Text #8

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.5305. Min Act: -0.0000

Data Index: 815555 (C4 (Web Text))

Max Activating Token Index: 1022

Text #9

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.8073. Min Act: -0.0000

Data Index: 551360 (C4 (Web Text))

Max Activating Token Index: 745

Text #10

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.4799. Min Act: -0.0000

Data Index: 1313518 (C4 (Web Text))

Max Activating Token Index: 403

Text #11

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.4163. Min Act: -0.0000

Data Index: 1085710 (C4 (Web Text))

Max Activating Token Index: 629

Text #12

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.2477. Min Act: -0.0000

Data Index: 340871 (C4 (Web Text))

Max Activating Token Index: 317

Text #13

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.2353. Min Act: -0.0000

Data Index: 1098017 (C4 (Web Text))

Max Activating Token Index: 752

Text #14

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.2638. Min Act: -0.0000

Data Index: 692110 (C4 (Web Text))

Max Activating Token Index: 326

Text #15

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.1113. Min Act: -0.0000

Data Index: 468440 (C4 (Web Text))

Max Activating Token Index: 612

Text #16

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.0993. Min Act: -0.0000

Data Index: 136266 (C4 (Web Text))

Max Activating Token Index: 679

Text #17

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.3298. Min Act: -0.0000

Data Index: 320282 (C4 (Web Text))

Max Activating Token Index: 635

Text #18

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.3271. Min Act: -0.0000

Data Index: 1012010 (C4 (Web Text))

Max Activating Token Index: 61

Text #19

Max Range: 3.2623. Min Range: -3.2623

Max Act: 2.1331. Min Act: -0.0000

Data Index: 333595 (C4 (Web Text))

Max Activating Token Index: 557