Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 3606 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 3.4964. Min Range: -3.4964

Max Act: 3.4964. Min Act: -0.0000

Data Index: 417793 (The Pile)

Max Activating Token Index: 324

Text #1

Max Range: 3.4964. Min Range: -3.4964

Max Act: 2.4782. Min Act: -0.0000

Data Index: 1166384 (The Pile)

Max Activating Token Index: 603

Text #2

Max Range: 3.4964. Min Range: -3.4964

Max Act: 2.3087. Min Act: -0.0000

Data Index: 152747 (The Pile)

Max Activating Token Index: 551

Text #3

Max Range: 3.4964. Min Range: -3.4964

Max Act: 2.7152. Min Act: -0.0000

Data Index: 772809 (The Pile)

Max Activating Token Index: 712

Text #4

Max Range: 3.4964. Min Range: -3.4964

Max Act: 2.4265. Min Act: -0.0000

Data Index: 1515857 (The Pile)

Max Activating Token Index: 579

Text #5

Max Range: 3.4964. Min Range: -3.4964

Max Act: 2.3329. Min Act: -0.0000

Data Index: 703765 (The Pile)

Max Activating Token Index: 650

Text #6

Max Range: 3.4964. Min Range: -3.4964

Max Act: 2.2793. Min Act: -0.0000

Data Index: 1745394 (The Pile)

Max Activating Token Index: 37

Text #7

Max Range: 3.4964. Min Range: -3.4964

Max Act: 1.8896. Min Act: -0.0000

Data Index: 1486897 (The Pile)

Max Activating Token Index: 815

Text #8

Max Range: 3.4964. Min Range: -3.4964

Max Act: 2.0847. Min Act: -0.0000

Data Index: 610717 (The Pile)

Max Activating Token Index: 464

Text #9

Max Range: 3.4964. Min Range: -3.4964

Max Act: 1.9462. Min Act: -0.0000

Data Index: 1176988 (The Pile)

Max Activating Token Index: 772

Text #10

Max Range: 3.4964. Min Range: -3.4964

Max Act: 2.4505. Min Act: -0.0000

Data Index: 1558261 (The Pile)

Max Activating Token Index: 821

Text #11

Max Range: 3.4964. Min Range: -3.4964

Max Act: 2.1540. Min Act: -0.0000

Data Index: 244047 (The Pile)

Max Activating Token Index: 293

Text #12

Max Range: 3.4964. Min Range: -3.4964

Max Act: 2.2334. Min Act: -0.0000

Data Index: 506741 (The Pile)

Max Activating Token Index: 920

Text #13

Max Range: 3.4964. Min Range: -3.4964

Max Act: 1.8189. Min Act: -0.0000

Data Index: 1706375 (The Pile)

Max Activating Token Index: 827

Text #14

Max Range: 3.4964. Min Range: -3.4964

Max Act: 2.0130. Min Act: -0.0000

Data Index: 1150068 (The Pile)

Max Activating Token Index: 986

Text #15

Max Range: 3.4964. Min Range: -3.4964

Max Act: 2.0896. Min Act: -0.0000

Data Index: 1108200 (The Pile)

Max Activating Token Index: 440

Text #16

Max Range: 3.4964. Min Range: -3.4964

Max Act: 2.0942. Min Act: -0.0000

Data Index: 1247739 (The Pile)

Max Activating Token Index: 159

Text #17

Max Range: 3.4964. Min Range: -3.4964

Max Act: 1.9966. Min Act: -0.0000

Data Index: 433684 (The Pile)

Max Activating Token Index: 831

Text #18

Max Range: 3.4964. Min Range: -3.4964

Max Act: 1.9458. Min Act: -0.0000

Data Index: 1924749 (The Pile)

Max Activating Token Index: 416

Text #19

Max Range: 3.4964. Min Range: -3.4964

Max Act: 1.9584. Min Act: -0.0000

Data Index: 1633516 (The Pile)

Max Activating Token Index: 589