Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4449 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.2625. Min Range: -2.2625

Max Act: 2.2123. Min Act: -0.0000

Data Index: 639568 (C4 (Web Text))

Max Activating Token Index: 697

Text #1

Max Range: 2.2625. Min Range: -2.2625

Max Act: 2.1741. Min Act: -0.0000

Data Index: 201844 (C4 (Web Text))

Max Activating Token Index: 484

Text #2

Max Range: 2.2625. Min Range: -2.2625

Max Act: 2.0007. Min Act: -0.0000

Data Index: 507792 (C4 (Web Text))

Max Activating Token Index: 106

Text #3

Max Range: 2.2625. Min Range: -2.2625

Max Act: 2.1062. Min Act: -0.0000

Data Index: 535908 (C4 (Web Text))

Max Activating Token Index: 128

Text #4

Max Range: 2.2625. Min Range: -2.2625

Max Act: 2.2625. Min Act: -0.0000

Data Index: 1237048 (C4 (Web Text))

Max Activating Token Index: 606

Text #5

Max Range: 2.2625. Min Range: -2.2625

Max Act: 2.0022. Min Act: -0.0000

Data Index: 441329 (C4 (Web Text))

Max Activating Token Index: 632

Text #6

Max Range: 2.2625. Min Range: -2.2625

Max Act: 2.0955. Min Act: -0.0000

Data Index: 470064 (C4 (Web Text))

Max Activating Token Index: 366

Text #7

Max Range: 2.2625. Min Range: -2.2625

Max Act: 2.0798. Min Act: -0.0000

Data Index: 1119691 (C4 (Web Text))

Max Activating Token Index: 543

Text #8

Max Range: 2.2625. Min Range: -2.2625

Max Act: 2.0617. Min Act: -0.0000

Data Index: 213294 (C4 (Web Text))

Max Activating Token Index: 564

Text #9

Max Range: 2.2625. Min Range: -2.2625

Max Act: 2.0041. Min Act: -0.0000

Data Index: 390363 (C4 (Web Text))

Max Activating Token Index: 826

Text #10

Max Range: 2.2625. Min Range: -2.2625

Max Act: 2.0587. Min Act: -0.0000

Data Index: 1256488 (C4 (Web Text))

Max Activating Token Index: 501

Text #11

Max Range: 2.2625. Min Range: -2.2625

Max Act: 2.0791. Min Act: -0.0000

Data Index: 304951 (C4 (Web Text))

Max Activating Token Index: 898

Text #12

Max Range: 2.2625. Min Range: -2.2625

Max Act: 1.5570. Min Act: -0.0000

Data Index: 399722 (C4 (Web Text))

Max Activating Token Index: 554

Text #13

Max Range: 2.2625. Min Range: -2.2625

Max Act: 1.6410. Min Act: -0.0000

Data Index: 177003 (C4 (Web Text))

Max Activating Token Index: 210

Text #14

Max Range: 2.2625. Min Range: -2.2625

Max Act: 1.6258. Min Act: -0.0000

Data Index: 41881 (C4 (Web Text))

Max Activating Token Index: 394

Text #15

Max Range: 2.2625. Min Range: -2.2625

Max Act: 1.6108. Min Act: -0.0000

Data Index: 1531470 (Python Code)

Max Activating Token Index: 455

Text #16

Max Range: 2.2625. Min Range: -2.2625

Max Act: 1.4924. Min Act: -0.0000

Data Index: 637125 (C4 (Web Text))

Max Activating Token Index: 1012

Text #17

Max Range: 2.2625. Min Range: -2.2625

Max Act: 1.4924. Min Act: -0.0000

Data Index: 915089 (C4 (Web Text))

Max Activating Token Index: 77

Text #18

Max Range: 2.2625. Min Range: -2.2625

Max Act: 1.4885. Min Act: -0.0000

Data Index: 684031 (C4 (Web Text))

Max Activating Token Index: 823

Text #19

Max Range: 2.2625. Min Range: -2.2625

Max Act: 1.4697. Min Act: -0.0000

Data Index: 628632 (C4 (Web Text))

Max Activating Token Index: 755