Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 854 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.6386. Min Act: -0.0000

Data Index: 1625754 (The Pile)

Max Activating Token Index: 15

Text #1

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.5407. Min Act: -0.0000

Data Index: 1310692 (The Pile)

Max Activating Token Index: 34

Text #2

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.3392. Min Act: -0.0000

Data Index: 1225963 (The Pile)

Max Activating Token Index: 64

Text #3

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.2095. Min Act: -0.0000

Data Index: 1561747 (The Pile)

Max Activating Token Index: 889

Text #4

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.2650. Min Act: -0.0000

Data Index: 1325627 (The Pile)

Max Activating Token Index: 582

Text #5

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.1975. Min Act: -0.0000

Data Index: 1084295 (The Pile)

Max Activating Token Index: 812

Text #6

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.2579. Min Act: -0.0000

Data Index: 742324 (The Pile)

Max Activating Token Index: 71

Text #7

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.2578. Min Act: -0.0000

Data Index: 364822 (The Pile)

Max Activating Token Index: 385

Text #8

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.2588. Min Act: -0.0000

Data Index: 600145 (The Pile)

Max Activating Token Index: 280

Text #9

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.0485. Min Act: -0.0000

Data Index: 827687 (The Pile)

Max Activating Token Index: 342

Text #10

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.2480. Min Act: -0.0000

Data Index: 1553933 (The Pile)

Max Activating Token Index: 756

Text #11

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.1680. Min Act: -0.0000

Data Index: 1846418 (The Pile)

Max Activating Token Index: 678

Text #12

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.1449. Min Act: -0.0000

Data Index: 1890812 (The Pile)

Max Activating Token Index: 883

Text #13

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.0918. Min Act: -0.0000

Data Index: 1777802 (The Pile)

Max Activating Token Index: 537

Text #14

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.2197. Min Act: -0.0000

Data Index: 774947 (The Pile)

Max Activating Token Index: 533

Text #15

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.1481. Min Act: -0.0000

Data Index: 1031208 (The Pile)

Max Activating Token Index: 212

Text #16

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.1955. Min Act: -0.0000

Data Index: 431019 (The Pile)

Max Activating Token Index: 725

Text #17

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.1308. Min Act: -0.0000

Data Index: 1118160 (The Pile)

Max Activating Token Index: 932

Text #18

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.0906. Min Act: -0.0000

Data Index: 641043 (The Pile)

Max Activating Token Index: 128

Text #19

Max Range: 1.6386. Min Range: -1.6386

Max Act: 1.1942. Min Act: -0.0000

Data Index: 1133179 (The Pile)

Max Activating Token Index: 678